AP-Fail

[MontanaMike]

Good morning,

Since upgrading to 7.4.3 on my 601E firewall cluster, my AP's (only 9 of them) keep dropping offline with no good explanation in the logs that I can tell.  I never had a problem with the previous version I was at which was 7.4.1.  A reboot of the AP (either by resetting the POE on the switch port or by unplugging and plugging back in) will bring the AP back online and connected with clients but then randomly (could be hours or days later) it will drop back off.

My Fortigate cluster is stable (no HA changes) and seems to be normal.  The APs are all 221Es running 7.4.2 firmware.  The only log entry I see that seems to be related to when they drop is:

Action ap-fail
Reason Control message maximal retransmission limit reached

Profile resv-dflt-FP221E5519035229
Physical AP ap-2b-public
Mesh Mode mesh root ap
Message Failure happened on AP ap-2b-public.

I did find a document (https://community.fortinet.com/t5/FortiAP/Troubleshooting-Tip-After-a-failover-FortiAP-devices-fail-...) and increased the timeout on the Fortigates so we'll see how that goes but I didn't have to do that on the previous versions.

# config wireless-controller global

set max-retransmit 15

# config wireless-controller timers

set echo-interval 100

Any help would be much appreciated. 

[hbac · Accepted solution]

Hi @MontanaMike,

It seems to match a known bug ID 0955764. However, you need to open a ticket to verify if the matches or not. 

Regards, 

[MontanaMike]

Do you have a link to the description?

[hbac]

@MontanaMike,

You can refer to the link below and look for Bug ID 998578. 

https://docs.fortinet.com/document/fortigate/7.4.3/fortios-release-notes/236526/known-issues

Regards, 

[MontanaMike]

Fortinet Support got back to me and confirmed the bug.

"Dear Customer,
   Thanks for contacting fortinet. I am looking into this ticket and will be happy to assist you with it. 

With regards to the issue you are seeing, this is a known issue tracked under bug 0955764, where fap 221Es are losing connection to fgt on 7.4.2/7.4.3.  Engineering has looked into this issue and they have been able to root cause. The issue will be addressed in 7.4.4 fortigate/fortiOS release. ETA for 7.4.4 is around 3rd week of April, 2024.

Engineering has suggested either of the below workarounds for now.

1 Downgrade of the fgt to 7.4.1 release.
2  OR rebooting the APs which are seeing the issue to bring the APs back online.

Please let me know for anything.

Thanks and regards,"

[MontanaMike]

Interesting thing is of the 9 APs I have attached to the Fortigate cluster, only about 1/2 of them keep dropping off and have to be rebooted.  I've increased the timeout on the Fortigate and have checked the physical layer for any issues which appear to be fine.  All the APs are the same model (221E) so I'm curious as to why only 1/2 drop off.  I don't think it's traffic either because when they do, it's usually when no one is around to connect to them.  i.e. the middle of the night.

[hpdcomputercenter]

Thanks! So glad I found this thread. We've been having the exact same issue with our 224Es. Every day I come in, and several of them are offline. I think I'll wait for the new FortiOS rather than downgrade.

[MontanaMike]

One thing that has helped me is creating an automation notification for when the AP "leaves" and "joins" so I get an alert when it happens.  If I happen to be remote I can log into the POE switch and reset (or turn off then on) the POE for the port of the AP affected and that essentially reboots my APs.  I do have a couple APs that are on a non-poe switch using injectors so those have to be manually (unplug, plug in) rebooted.

hope they come out with the updated firmware soon.

[Adolfo_Z_H]

Hi Mike, i am the Author of the community article you mentioned.

Those values are examples of what happens when you change it, not a suggestions.

suggestions are follows

In summary, to speed up FortiAP reconnections:

1) Use the default values on these timers where possible.

2) Use manual controller discovery and manual IP addressing on the APs.

It is no longer required to change these timer settings from their default values on modern high speed, high bandwidth networks.

If FortiAP failures and disconnections occur with the following message...

'ECHO REQ is missing' and 'Control message maximal retransmission limit reached'

... And the related APs are deployed as local FortiAPs (they are on the same campus, typically in the same building, with gigabit speed links or better), consider investigating for Network issues or FortiAP related issues before attempting to tune up wireless controller timers and global settings. The default settings are recommended for most deployments.

Read the following article to understand how to diagnose FortiAP related issues:

https://community.fortinet.com/t5/FortiAP/Technical-Tip-How-to-interpret-FortiWiFi-or-FortiAP-variou...

References

https://docs.fortinet.com/document/fortigate/6.2.3/cli-reference/138620/wireless-controller-timers

https://docs.fortinet.com/document/fortigate/6.0.0/cli-reference/717332/wireless-controller-timers

https://docs.fortinet.com/document/fortigate/6.0.0/cli-reference/214787/wireless-controller-global

https://community.fortinet.com/t5/FortiAP/Technical-Tip-How-to-interpret-FortiWiFi-or-FortiAP-variou...

https://docs.fortinet.com/document/fortiap/7.0.4/fortiwifi-and-fortiap-configuration-guide/65088/for...

[MontanaMike]

Interesting to note:  7.4.4 has been out and there is no mention of bug ID 0955764 in the release notes.  I chatted with support and they said that engineering must not have fix the issue however after applying 7.4.4 to my main cluster I haven't had a 221E AP disconnect.  It's been almost 2 weeks and so far, so good.