FortiNAC-F 7.2.5, New core switch, Contact Lost issues

[nkuhl30]

We replaced our core switch (HP 8212zl) with an Aruba CX 6405 back in June. We hired a consultant to reconfigure the config to be compatible with the new CX OS. Ever since then, FortiNAC is intermittently reporting Contact Lost events for our Aruba Controllers and Aruba APs. The controllers and the APs are on the same VLAN, and there are no ACLs or firewalls in-between VLAN 1 (new core) and VLAN 40 (wireless network).

There are zero port errors on the core and it's not an STP issue. The contact lost events only occur with the wireless controllers and APs, and not with our other edge switches or servers.

If we check the Aruba wireless side of things, none of the controllers or APs lose network connectivity. There's just a brief/random comms issue when FortiNAC reaches out to poll via ICMP.

Does anyone have any ideas? Fortinet support is claiming it's a network issue and not on their end. I could really use some help.

Thank you.

[ndumaj]

Hi @nkuhl30 

Can you run a PCAP on the FW where FNAC and WLC are connected to confirm that there is a connection issue?

From FNAC you can easily run PCAP:
execute tcpdump -i any host <WLC IP> -w WLC.pcap

Article:
https://community.fortinet.com/t5/FortiNAC-F/Technical-Tip-Run-tcpdump-in-FNAC-F-and-save-capture-as...

BR

[nkuhl30]

This is extremely helpful. I'll do that and hopefully it points the way to a resolution. 

I went back and forth with TAC on this and they never mentioned doing this. Ugh.

[ndumaj]

Waiting for your feedback.

We need to identify if the FNAC is contacting the WLC? Yes, NO?
Yes is the WLC receiving the request? Yes, NO?
Is WLC responding to the FNAC request? Yes, NO?
Is the FNAC receiving the response? Yes, NO?

BR

[nkuhl30]

Thanks. The packet capture it running. I just need to wait until it occurs again. BTW,

what's the CLI command to delete the PCAP file from FortiNAC? 

[ndumaj]

Hello,

You can use "rm" and the <file name>
or instead of scp command, you can use mv to export the outside of the box.

BR

[nkuhl30]

Thanks, so it just happened and the packet capture got it. 10.0.1.14 is FortiNAC and 10.40.0.11 is one of our Aruba wireless controllers.

We can see that FortiNAC is sending the ICMP request but not receiving a reply for about a minute. One thing I did notice is a lot of fragmented frames earlier as the the SNMP communication is going on. That doesn't look right.

Any thoughts or suggestions?

[AEK]

I guess you are allowing jumbo frames between WLC and Aruba switch, and somewhere on the path to FNAC it is set to 1500 again.

I remember network guys don't like fragmentation, I guess for its CPU overhead on the switch. If this is really the case then I'd fix this (set 1500 along all the path) and redo the test.

[nkuhl30]

That's the thing, jumbo's aren't enabled on our new core switch. The default MTU on Aruba CX is 1500. 

We moved from an HP Procurve core switch to Aruba CX back in June and this started happening immediately, but only with the Aruba wireless controllers and APs. The default MTU of the HP Procurve core is 1522. That's the only difference that I can see. The configs are the same.

[AEK]

So set it to 1522 as it was before and test again :)

[nkuhl30]

So, dumb question, should I try setting the MTU on the new core interface or VLAN 1522?