Fortigate suddenly stops commincating with Forticollector/Fortianalyzer

[Scott2]

Hello,

Strange problem with connectivity between FTG and FAZ. which I have encountered before. The fortigate unit 100d & a 600c (currently running 5.0.6 firmware), suddenly just stops logging to Fortianalyzer, and this is what alerts us to the issue. But the issue might not be logging itself.

Its still trying to send longs, as I can see them being generated in a sniffer output. But for some reason despite route being in place it loses ping connectivity to Fortianalyzer, yet can ping another device in exactly same subnet, in this case the Fortimanager on our management network. Can still retrieve configuration, push policy, and connect to other parts of the network, only ever seems to lose ping connectivity to the FAZ.

Seen this before with another device in the past, and only option ended up being to reboot the FTG, but wondering if there is anything else I can do to diagnose/trouble shoot and potentially fix what appears to be some sort of route/arp corruption/issue, rather than extreme of rebooting device completely. To restore connectivity to the FAZ and get my logging working again.

Anyone seen this before, any advice greatly appreciated.

Kind regards

Scott

[iJake]

This has happened to both the FortiGates? It's a strange thing to happen out of the blue.

Have you tried doing a sniffer on the FAZ to see if it's receiving this traffic?

EDIT: It's not screaming ARP issue with the FortiGates if the traffic is getting routed to the FAZ and you can ping other devices on that subnet.

[Scott2]

Hello,

Yes happened to both units in different locations, one of them it has happened to before, the 600c (reboot device and works perfectly again).

When running the sniffer on FTG and FAZ, I can see the FTG generating the UDP 514 log traffic, but never gets to FAZ.

Which make sense, as I know I have confirmed, I cannot ping FAZ/Collector. Unable to trace to it or ping it. Yet I can ping an address on same subnet / network as the FAZ, literally the IPs that I can reach can be side by side, to that of the FAZ which I cannot reach.

Also from next hop/routed path, which is a core switch I am able to ping FAZ/collector perfectly fine. There are no ACL's / policies in place that would prevent traffic being sent and there has been no changes to device.

As I say it just suddenly will not ping the FAZ, and again only ever appears to lose connectivity with the FAZ, nothing else, hence why I decided to post in the logging forum.

I know a reboot will fix it, but just seems strange that it would suddenly do this out of the blue.

Kind regards

Scott

[Scott2]

Thinking possibly something firmware related as we are still running quite old code 5.0.6. But we are planning to upgrade shortly to 5.2.1 or maybe even 5.2.2 when it comes out.

But would like to understand if it is a firmware issue or something else that is going on. One of those niggles that you think, I really want to know why and make sure the actions I take are the appropriate ones, rather than take action hoping its going to fix it.

Kind regards

Scott

[iJake]

That is very strange behavior

It's certainly better to try and resolve the issue that just rebooting to hope it fixes it.

If it is an ARP issue, you can check by running "get system arp" or clear it by running "exec clear system arp table" but I would only attempt this if it's essential to get logging back, otherwise it's better to find the cause.

[Scott2]

This is how strange the issue is, in this location we have two collectors , and one analyzer (as we split this by region to share load of log collection, and then send logs onto analyzer in each location).

10.111.164.24 is the analyzer

10.111.164.25 is the collector this unit should log too

10.111.164.26 is another collector in same region

This is the Fortianalyzer settings pointing it to the .25 collector:-

# config log fortianalyzer setting

 (setting) # show config log fortianalyzer setting set status enable set server 10.111.164.25 set upload-option realtime end

Ping to collector it should log to does not work.

# execute ping 10.111.164.25 PING 10.111.164.25 (10.111.164.25): 56 data bytes

--- 10.111.164.25 ping statistics --- 5 packets transmitted, 0 packets received, 100% packet loss

Pings to other collector/analyser literally single digits away in same VLAN/Subnet

 # execute ping 10.111.164.24 PING 10.111.164.24 (10.111.164.24): 56 data bytes 64 bytes from 10.111.164.24: icmp_seq=0 ttl=60 time=100.9 ms 64 bytes from 10.111.164.24: icmp_seq=1 ttl=60 time=39.5 ms 64 bytes from 10.111.164.24: icmp_seq=2 ttl=60 time=78.4 ms 64 bytes from 10.111.164.24: icmp_seq=3 ttl=60 time=40.7 ms 64 bytes from 10.111.164.24: icmp_seq=4 ttl=60 time=77.8 ms

--- 10.111.164.24 ping statistics --- 5 packets transmitted, 5 packets received, 0% packet loss round-trip min/avg/max = 39.5/67.4/100.9 ms

# execute ping 10.111.164.26 PING 10.111.164.26 (10.111.164.26): 56 data bytes 64 bytes from 10.111.164.26: icmp_seq=0 ttl=60 time=53.3 ms 64 bytes from 10.111.164.26: icmp_seq=1 ttl=60 time=43.7 ms 64 bytes from 10.111.164.26: icmp_seq=2 ttl=60 time=41.9 ms 64 bytes from 10.111.164.26: icmp_seq=3 ttl=60 time=38.7 ms 64 bytes from 10.111.164.26: icmp_seq=4 ttl=60 time=39.9 ms

--- 10.111.164.26 ping statistics --- 5 packets transmitted, 5 packets received, 0% packet loss round-trip min/avg/max = 38.7/43.5/53.3 ms

Again no policies or devices in the way that would be blocking any of this traffic. Routing is done either at the /16 or /24 bit boundary's of the subnets and confirmed they are there, and was working previously, just stops working suddenly, and only reboot seems to fix issue.

Kind regards

Scott

[iJake]

Can other devices on the same subnet as the FortiGates ping the collector at .24?

If it's something you can wait for, it is probably worth opening a ticket with FortiNet, if not and you think it's ARP, use the command I posted above, to see if you can avoid a reboot.