Fortiswitch | ISL Trunk disconnections

[sam_hammoud]

Hi,

I have been trying to resolve an instability issue with our Fortiswitch stack with support. They have been unable to conclusively fix the issue, however through a process of elimination we have been able regain some stability in the stack.

It looks like Sonos (which unfortunately I can't get rid of) was causing some issues, we think this is due to SonosNet. An issue very similar to this: https://www.verschoren.com/2017/04/sonos-network-issues/

Now we seem to have two members in the stack that seem to drop off randomly either together or individually, this is happening 3-4 times a day. This results in all devices connected to the switch dropping their connections and dropping 5 to 15 packets. I have jumped onto one of switches and this is what is in the log when it drops:

134: 2017-12-18 13:52:33 log_id=0100033142 type=event subtype=system pri=notice vd=root action=switch-config-set status=connected msg="Switch-Controller: connected with FortiGate"

135: 2017-12-18 13:52:32 log_id=0100020202 type=event subtype=system pri=information vd=root action=daemon-startup daemon=cu_swtpd pid=2417 msg="Daemon cu_swtpd started"

136: 2017-12-18 13:52:31 log_id=0100020203 type=event subtype=system pri=information vd=root action=daemon-shutdown daemon=cu_swtpd pid=2237 msg="Daemon cu_swtpd shut down"

137: 2017-12-18 13:52:31 log_id=0100032141 type=event subtype=system pri=information vd=root id=30 msg="interface internal gets a DHCP lease, ip:169.254.1.7, mask:255.255.255.0, gateway:169.254.1.1, lease expires:Mon Dec 25 13:52:27 2017 dns1:0.0.0.0 dns2:0.0.0.0 change:0"

138: 2017-12-18 13:52:20 log_id=0100033143 type=event subtype=system pri=notice vd=root msg="FortiLink: ISL timing-out for trunk(8DF3X16000745-0) member port(51) did not receive ISL pkt for(10) sec"

139: 2017-12-18 13:52:09 log_id=0100033143 type=event subtype=system pri=warning vd=root msg="FortiLink: internal echo reply timed out"

140: 2017-12-18 13:51:54 log_id=0100033143 type=event subtype=system pri=notice vd=root msg="FortiLink: ISL timing-out for trunk(8DF3X16000855-0) member port(49) did not receive ISL pkt for(10) sec"

141: 2017-12-18 13:51:52 log_id=0100033143 type=event subtype=system pri=warning vd=root msg="FortiLink: internal echo reply timing out echo-miss(20)"

142: 2017-12-18 13:51:38 log_id=0100033143 type=event subtype=system pri=warning vd=root msg="FortiLink: internal echo reply timing out echo-miss(10)"

143: 2017-12-18 13:50:54 log_id=0100033143 type=event subtype=system pri=notice vd=root msg="FortiLink: ISL timing-out for trunk(8DF3X16000745-0) member port(51) did not receive ISL pkt for(10) sec"

144: 2017-12-18 13:48:46 log_id=0100033143 type=event subtype=system pri=notice vd=root msg="FortiLink: ISL timing-out for trunk(8DF3X16000745-0) member port(51) did not receive ISL pkt for(10) sec"

145: 2017-12-18 13:44:20 log_id=0100033143 type=event subtype=system pri=notice vd=root msg="FortiLink: ISL timing-out for trunk(8DF3X16000745-0) member port(51) did not receive ISL pkt for(10) sec"

146: 2017-12-18 13:44:04 log_id=0100033143 type=event subtype=system pri=notice vd=root msg="FortiLink: ISL timing-out for trunk(8DF3X16000855-0) member port(49) did not receive ISL pkt for(10) sec"

147: 2017-12-18 13:43:52 log_id=0100033143 type=event subtype=system pri=notice vd=root msg="FortiLink: ISL timing-out for trunk(8DF3X16000745-0) member port(51) did not receive ISL pkt for(10) sec"

148: 2017-12-18 13:43:46 log_id=0100033143 type=event subtype=system pri=notice vd=root msg="FortiLink: ISL timing-out for trunk(8DF3X16000855-0) member port(49) did not receive ISL pkt for(10) sec"

149: 2017-12-18 13:43:20 log_id=0100033143 type=event subtype=system pri=notice vd=root msg="FortiLink: ISL timing-out for trunk(8DF3X16000855-0) member port(49) did not receive ISL pkt for(10) sec"

150: 2017-12-18 13:41:19 log_id=0100033143 type=event subtype=system pri=notice vd=root msg="FortiLink: ISL timing-out for trunk(8DF3X16000745-0) member port(51) did not receive ISL pkt for(10) sec"

151: 2017-12-18 13:41:05 log_id=0100033143 type=event subtype=system pri=notice vd=root msg="FortiLink: ISL timing-out for trunk(8DF3X16000745-0) member port(51) did not receive ISL pkt for(10) sec"

There is also an NTP error that appears around these events:

66: 2017-12-18 14:17:59 log_id=0100032140 type=event subtype=system pri=notice vd=root user="ntp_daemon" field=date-time msg="The ntp server 208.91.113.71 is determined unreachable at Mon Dec 18 14:17:59 2017
"

67: 2017-12-18 14:17:59 log_id=0100032140 type=event subtype=system pri=notice vd=root user="ntp_daemon" field=date-time msg="The ntp server 169.254.1.1 is determined unreachable at Mon Dec 18 14:17:59 2017
"

68: 2017-12-18 14:17:55 log_id=0100033143 type=event subtype=system pri=warning vd=root msg="FortiLink: internal echo reply timed out"

69: 2017-12-18 14:17:42 log_id=0100033143 type=event subtype=system pri=warning vd=root msg="FortiLink: internal echo reply timing out echo-miss(20)"

70: 2017-12-18 14:17:30 log_id=0100033143 type=event subtype=system pri=warning vd=root msg="FortiLink: internal echo reply timing out echo-miss(10)"

[bmduncan34]

Those NTP messages are known issues.  The FortiSwitches are pre-programmed to try and go to public NTP servers to get time, but only as a backup.  They actually get time syncs from the gate.  This post is three years old so I reckon you already figure that out.