Skip to main content
menatwork
menatwork
Visitor III
September 21, 2022
Question

Fortigate memory exhausted in seconds - 99% and not reacting to anything

  • September 21, 2022
  • 4 replies
  • 3762 views

Hi folks,

we have got a Fortigate 60F for 2 years now. It is running on FortiOS 7.0.6. Out of nowhere (about a week ago) it started to go in conserve mode. And even worse - after hitting 99% of mem usage, it does not react to anything. If it goes up to 99% you also cant use the cli anymore.

 

This also happened at night, where only servers are online.

 

About 3-10 Minutes after hitting the 99% mem, the usage drops again (to normal (for us) 63%) and it runs without troubles for hours.

 

It was a pain in the .... to investigate where it is coming from.

 

At least I figured out, that the problem is triggered by servers of our DMZ. (after the FGT starts to increase mem usage I disconnected all Ports and reconnected them one by one. LAN, WAN1, WAN2 does not trigger it, but reconnecting DMZ does).

 

After that I shut down all not essential servers running at DMZ. After the shutdown, the problem does not reappear again.

 

At the moment, it looks like that something is going on on the servers which are shut down at the moment. Something which kills the Forti.

 

I cannot see any suspicious traffic in the logs.

 

So, how should I approach this problem. Do you have any tipps? What could trigger the Fortigate that way, that the mem-usage goes up from 65% to 99% within 30 secs?

 

Thanks!

4 replies

menatwork
menatworkAuthor
Visitor III
September 21, 2022

just a little update: It seems like it is originating from our Nextcloudserver. As soon as it is fired up again, mem usage rises like said above. Nothing "special" to see in FGT-Logs. Weird...

menatwork
menatworkAuthor
Visitor III
September 21, 2022

so just another info. This is no Fortigate Issue. I did some sniffing and tcpdumping now (as I know where to look for).

 

I got 4000 packets in 3 seconds (sniffer @ forti). If you look at it, it seems like there are an enormous amount of requests from one ip adress (sitting in San-Francisco). I am no network-guru, but this seem to be just requests.

 

May this be some DOS-Attack?

bpozdena_FTNT
Staff
bpozdena_FTNT
Staff
September 21, 2022

If you see a high session setup rate from just a single IP address, you can quickly create a firewall policy for that specific source IP address and set the policy action to Deny. That alone should prevent session creation and therefore prevent memory utilization increase.

 

The Fortigate CPU might still suffer though. If that is the case, you can enable creation of denied sessions, which can help reduce CPU utilization caused by route and policy lookups.  More details at https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-enable-denied-session-to-be-added-into-the/ta-p/195478 .

 

 

 

With that said, a more permanent solution would be to configure a DoS policy and limit the amount of sessions a single source IP address can create. More details can be found at https://docs.fortinet.com/document/fortigate/7.0.6/administration-guide/771644/dos-protection .

 

FortiGate can also automatically quarantine the malicious source IP address. This just helps you gain a better visibility into the events on your network. Example can be found at https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-configure-DoS-protection-s-quarantine/ta-p/192590 .

 

 

menatwork
menatworkAuthor
Visitor III
September 22, 2022

@bpozdena_FTNT  Thank you very much for your answer. I did some more investigation. Its weird for me. Initially I thought that this may be an attack. After some additional "Wiresharking" I saw that the connection gets initiated from the inside. 

 

If it starts we get 4000 packets in about 5 seconds.

 

Looks like this: https://share-your-photo.com/3fa081c7ec

 

Its a clamav-database-Update (antivirus for Linux) connecting to database.clamav.net.cdn.cloudflare.net with an IP of 104.16.218.84. After that the packets run wild. 

 

atm I have no idea whats going on here, but as said above its no Fortigate issue.

bpozdena_FTNT
Staff
bpozdena_FTNT
Staff
September 22, 2022

If it starts we get 4000 packets in about 5 seconds.

A few Megabits per second of legitimate traffic should be of no concern. 

 

 

but as said above its no Fortigate issue.

Well, if your Fortigate runs out of memory, it is very much a Fortigate issue. 

 

Since the traffic appears to be legitimate, you will need to check where is the memory being consumed . The basic idea can be obtained by following these steps. 

 

As a quick workaround test, you can try to temporarily disable UTM on the affected firewall policies. 

 

However, if you haven't already, my suggestion would be to open a TAC ticket to get it properly looked into. If you can provide some of the outputs from the linked KB article when opening the ticket, it will speed up the process significantly. 

menatwork
menatworkAuthor
Visitor III
September 22, 2022

@bpozdena_FTNTnot only 4000 packets in 5 seconds. It got more and more and I am unsure if it is legetimate.

 

TAC Ticket is open for about 7 days now. (I upgraded firmware twice)

Terms & ConditionsAccessibility statement