"Memory conserve mode" every day at the exact same time.

[robertl]

Our fg100, running on 7.4.1, enters memory conserve mode every day at 4:41pm. 

Support has sat with us and watched it happen for multiple days, they gather all the logs they need each time. They offer no solutions, their only response is "we need more logs". 

If TAC has nothing better then I'm at a loss of what to do next.

[dbu]

Hi @robertl ,

Is there any scheduled activity corresponding with this memory usage?

It is strange happening at the exact time like programmed somehow. 
Perhaps some FortiGuard updates are done at the time you can double check. 

[robertl]

We, and your support techs, have checked and all scheduled activities occur between 10pm and 2am.

[dbu]

Is this a fresh install to 7.4.1 or you upgraded from earlier firmware release ? If Yes ,was this issue visible on the previous version ? 

[robertl]

It was upgraded from an earlier release. Started sometime after upgrade. We have 8 firewalls on this release and only 1 of them has this issue. Unfortunately, it is our outer firewall so basically brings down everything.

[hbac]

Hi @robertl,

Can you share the ticket number so I can take a look at the logs? 

Regards, 

[robertl]

8905087 Ticket Title: Memory Is Full

This ticket was opened and worked by our other sys admin so I'm not sure what details are in there. Thank you.

[robertl]

Watching it in real-time, there are a number of processes running named "ipsengine" and they usually run with a CPU load of 2%-3% each but at 4:41PM, the processes (8 of them) jump up to 20%-35% CPU usage each.. If I can quickly kill the processes once they start to climb I can prevent the memory conserve mode from triggering.

[robertl · Accepted solution]

Solved. I think, will find out tonight.

Our RMM tool (Datto) kicks off a job every day at 4:40PM to check for the latest version of Chrome and if not present, downloads the .msi and installs. Someone recently made a GPO to prevent Google Chrome updates. Since no endpoint is on the current version, this is 200+ machines downloading/installing an .msi. I think it's safe to assume the firewall is doing some inspection on all of these and causing our problem.