FortiAPs all drop at once with "Control message maximal retransmission limit reached"

[Ian_Bell]

Hi all - 

Had an incident on my network that I'm having trouble getting to the bottom of.

We have 7 FortiAPs (320c's) connecting to our Fortigate firewall as controller. They're on a private vLAN with no other devices. This morning all 7 disconnected with this pair of errors:

Jan 17 10:05:56 <local7.notice> date=2020-01-17 time=10:05:56 devname=SanJose-HQ devid=FG600B3911600212 logid=0104043553 type=event subtype=wireless level=notice vd="root" logdesc="Physical AP fail" sn="FP320C3X14007549" ap="S-WAP-03" profile="resv-dflt-FP320C3X14007549" ip=10.1.132.106 meshmode="mesh root ap" snmeshparent="N/A" action="ap-fail" reason="Control message maximal retransmission limit reached" msg="AP S-WAP-03 failed." Jan 17 10:05:56 <local7.notice> date=2020-01-17 time=10:05:56 devname=SanJose-HQ devid=FG600B3911600212 logid=0104043552 type=event subtype=wireless level=notice vd="root" logdesc="Physical AP leave" sn="FP320C3X14007549" ap="S-WAP-03" profile="resv-dflt-FP320C3X14007549" ip=10.1.132.106 meshmode="mesh root ap" snmeshparent="N/A" action="ap-leave" reason="Control message maximal retransmission limit reached" msg="AP S-WAP-03 left."

After about 10 minutes they reconnected and started function fine again. None of the APs had rebooted or lost power. What's interesting here (maybe) is that it doesn't appear to be a broadcast flood or anything causing packets to be dropped; instead for several seconds before the outage they all got ping deny responses from the controller, e.g.:

Jan 17 10:05:50 <local7.notice> date=2020-01-17 time=10:05:50 devname=SanJose-HQ devid=FG600B3911600212 logid=0001000014 type=traffic subtype=local level=notice vd=root srcip=10.1.132.106 srcintf="port3" dstip=10.1.132.1 dstintf="root" sessionid=1533065762 proto=1 action=deny policyid=0 dstcountry="Reserved" srccountry="Reserved" trandisp=noop service="PING" app="PING" duration=0 sentbyte=0 rcvdbyte=0 sentpkt=0 appcat="unscanned" crscore=30 craction=131072 crlevel=high

So, their heartbeat pings were getting denied by the default implicit deny rule (policyid=0) but there's no reason for that that  I can see. And it's also very odd to me that the problem then fixed itself without intervention. Any idea why the controller would suddenly stop accepting pings from its APs? We had no other network outage at the time, and the regular Fortigate firewall rules all continued functioning normally in the meantime.