[__cmdb_bg_fork:670] fork( ) failed: 12(Cannot allocate memory)

[MBR]

I experience issues on several different FortiGate 60E firewalls with firmware 5.4.2 and 5.4.3.

After a period of uptime I experience that new changes (for example a new created policy) aren't working.

When doing troubleshooting in the cli I get  the error "[__cmdb_bg_fork:670] fork( ) failed: 12(Cannot allocate memory)"

When checking memory usage at that time only 60% memory is used.

The firewalls are configured with a simple config. In basic only an allow out policy with some default AV,Application Control and SSL certificate inspection.

I have to reboot the unit to get it resolved for some time until the issues comes back.

Anyone experiencing the same issue or know if this is a know issue with the 5.4.2 and 5.4.3 releases and can confirm it is resolved in 5.4.4 or has a solution for this issue?

MBR

[Alex_talmage]

I've just come across this error and similar symptoms.

I'm running 5.4.2, and I've just set up a new interface running a dhcp server. Though the config appears to have stuck, it doesn't actually appear to have taken effect, I cannot ping the new interface, and the dhcp server doesn't appear to be functioning. When I ran diag debug enable, I'm seeing this exact message...

[Alex_talmage]

I've just worked around this:

diag sys top

Shows you the top processes. Hit M to sort by memory.

Process wad was consuming 2GB of memory. This runs cache, wan optimization.

Diag sys kill 11 <pid> killed the process. Error has stopped appearing in debug... I've got a ticket open with Fortinet so I will enquire as to why this was using so much resource and report back.

[emnoc]

Can you  try 5.4.4? But either way I would open a ticket w/FTNT and hope they give you  a speedy result.

Ken

[jaustgen]

Reporting that I just ran into this issue on a 60E running 5.4.3.

Change to source IP in firewall policy wasn't taking.  Went to start up debug flow and when I entered "diag debug enable", this error started popping up every second.

killed the most memory using instance of ipsengine, and the error went away, and my policy's new source IP worked. 

I then removed my test source IP from the policy, and in the CLI this error immediately resumed, and my policy change did not take.  Killed ipsengine again, error went away, policy change took OK. 

Unrelated, I found out 5.4.3 and 5.4.4 is hanging up 100E models, and support claims it's fixed in 5.4.5.  I'm going to start reverting everything I can to 5.4.1.

[peter_wickenberg]

I'm also having this problem on my Fortigate 30E, only way to access the firewall when this happens is through the serialport. But since it's continously sending this message to the console I'm unable to manage the firewall.

fork() failed

[__cmdb_bg_fork:668] fork( ) failed: 12(Cannot allocate memory)

fork() failed

[__cmdb_bg_fork:668] fork( ) failed: 12(Cannot allocate memory)

fork() failed

[__cmdb_bg_fork:668] fork( ) failed: 12(Cannot allocate memory)

.....

Before this happened I didn't even have that many active sessions (about 2500), and also it happened when I was editing a firewall policy.

[MBR]

Hi Peter,

Which FortiOS version are you using?

[peter_wickenberg]

MBR wrote:

Hi Peter,

Which FortiOS version are you using?

I'm running the latest 5.6 on my 30E. Just did a factory restore and then restored my configuration so now I'm back up and running, for a while at least. This has happened two times earlier, seems like when the internal storage gets filled up it sooner or later starts acting up, as if the flash memory was flawed and when it stumbles upon this the error starts. This is just me guessing the cause of the problem, but since there doesn't appear to be any utilities for controlling this it's difficult to diagnostic. Also I did format the entire flash and restored the 5.6 firmware using TFTP but even this didn't seem to solve it as the error re-appeared.

[Uwe_Sommerfeld]

I can report I saw this error on a FG-VM00 and VM01 on FortiOS 5.6.0 as well. 

Not much diagnosis, but the main memory consumer was the ipsengine process. 

The good thing with VMs is that you can ramp up the "hardware". With VM02 settings, the issue doesn't happen for the moment. 

Load ~4000 sessions, cpu nothing, memory ~75% (VM01).

[MBR]

Sadly FortiOS 5.4.4 also has this issue.

So the only hope for this moment is the 5.4.5 version.

There is nothing mentioned about this issue in the release notes however.

5.6.0 also has this issue and 5.6.1 is not released yet.