EMS 1.2.1: Machine hangs very often

[CoSax]

Our EMS is now version 1.2.1. Version 1.0.0 was initially installed and then upgraded to the newer version whenever it was available.

The problem we face now is quite severe. The machine hangs at least once a day and the only thing you can do is restart. I increased considerably both memory and CPU but no improvement. The EMS logs indicate the following errors:

[08-03 07:18:27][  ERROR]: [0x1aecb768] error:00000005:lib(0):func(0):DH lib [08-03 07:18:27][  ERROR]: [0x1aecb768] fatal_error_occurred, session_close [08-03 07:22:54][  ERROR]: [0x1ad969c0] BIO_pending fatal_error_occurred = true; 5 [08-03 07:22:54][  ERROR]: [0x1ad969c0] error:00000005:lib(0):func(0):DH lib [08-03 07:22:54][  ERROR]: [0x1ad969c0] fatal_error_occurred, session_close [08-03 07:24:42][  ERROR]: [0x1ada6c08] fatal_error_occurred, session_close [08-03 07:25:02][  ERROR]: [0x1b2466b8] fatal_error_occurred, session_close [08-03 07:25:28][  ERROR]: [0x1ac9ffd0] fatal_error_occurred, session_close [08-03 07:25:50][  ERROR]: [0x1ac9ffd0] fatal_error_occurred, session_close [08-03 07:26:00][  ERROR]: [0x1b19e1f8] fatal_error_occurred, session_close [08-03 07:26:06][  ERROR]: [0x1ae45108] fatal_error_occurred, session_close [08-03 07:26:12][  ERROR]: [0x1b2466b8] fatal_error_occurred, session_close [08-03 07:27:30][  ERROR]: [0x1b2df3a0] fatal_error_occurred, session_close [08-03 07:28:26][  ERROR]: [0x1af42de8] fatal_error_occurred, session_close [08-03 07:30:13][  ERROR]: [0x1af62d38] Error : 997, session_accept renew_session status : 1 [08-03 07:30:30][  ERROR]: [0x1ad969c0] fatal_error_occurred, session_close [08-03 07:30:57][  ERROR]: [0x1b26eb88] fatal_error_occurred, session_close [08-03 07:31:24][  ERROR]: [0x1ada6c08] fatal_error_occurred, session_close [08-03 07:31:35][  ERROR]: [0x1b1ae200] Error : 997, session_accept renew_session status : 1 [08-03 07:31:36][  ERROR]: [0x1b1ae200] fatal_error_occurred, session_close [08-03 07:31:37][  ERROR]: [0x1b2ef5e8] fatal_error_occurred, session_close [08-03 07:31:56][  ERROR]: [0x1af62d38] Error : 997, session_accept renew_session status : 1

In the event log the following errors appears quite often:

Log Name:      Application Source:        Application Error Date:          3/8/2017 7:32:02 πμ Event ID:      1000 Task Category: (100) Level:         Error Keywords:      Classic User:          N/A Computer:      Description: Faulting application name: FcmDaemon.exe, version: 1.2.1.394, time stamp: 0x595ebc77 Faulting module name: policyhelper.dll, version: 1.2.1.394, time stamp: 0x595ebc63 Exception code: 0xc0000005 Fault offset: 0x00021300 Faulting process id: 0x8f4 Faulting application start time: 0x01d30b794f4c3799 Faulting application path: C:\Program Files (x86)\Fortinet\FortiClientEMS\FcmDaemon.exe Faulting module path: C:\Program Files (x86)\Fortinet\FortiClientEMS\policyhelper.dll Report Id: b201e0ee-7804-11e7-8107-005056840414 Faulting package full name: Faulting package-relative application ID:

Log Name:      System Source:        Service Control Manager Date:          3/8/2017 7:32:26 πμ Event ID:      7031 Task Category: None Level:         Error Keywords:      Classic User:          N/A Computer:      Description: The FortiClient Enterprise Management Server service terminated unexpectedly.  It has done this 255 time(s).  The following corrective action will be taken in 2000 milliseconds: Restart the service.

Any ideas how to sort this out??

[jklems]

We have been playing around with our hyperV settings.  Our EMS server has been running 21 days without issues.

Our current hyperV settings for EMS 1.2.1.0394

C Drive - 100GB - 83.3GB free at the moment.

Startup RAM - 8192MB

Disabled Dynamic RAM

Virtual Processors  - 2

So far we have about 90 hosts connected to this thing.  For those of you with more, how many hosts are you seeing before your EMS gets bogged down , slow and eventually no longer lets you use it ?

[Carl_Wallmark]

We have around 1000 clients.

Our problem is that the FortiClient Management Service keeps restarting about every minute. The clients cannot download the profile or connect, which causes huge problems.

What versions do your clients use ?

[jklems]

1000 Clients!  Nice!  well, were hoping to replace our 500 clients of Kaspersky with Forticlient 5.6, so far we just rolled 5.6 out to our laptop  / Surface  / and Macbook users.  We even have a few legacy XP workstations (dont ask) running 5.4.4 connecting to us via IPSec VPN.  We Also run a Fortigate 500d at our HQ.  and smaller Fortigates for small branches.  We did have to do some tweaking on the Fortigate side (changed telemetry listening port) so that Forticlient users didnt register with Fortigate only instead of the EMS server.   So now, our Fortigates all run 5.6 and almost all Forticlient users are on 5.6 which seems to run smoother then it used to.  21 days is a new record for sure!  But I imagine 1000 users puts a heavier load on EMS.  Do you guys run multiple groups? multiple profiles ? so that all clients arent trying to run vulnerability scans and AV scans at the same time ?  

[CoSax]

We have 1300+ clients so far with similar problems. EMS hangs completely every 1.5 hours (100% CPU) during working hours. Impossible to work with.

[jklems]

just curious if you EMS server is on vmware or hyperV.  are you using dynamic vs static memory.  also curious how many virtual cores you assigned to the virtual machine.  What OS ?  in my case, Windows server 2012 standard running on HyperV

[CoSax]

I have ridiculously high specs in order to make it run for few hours:

windows server 2012R2 on Vmware, 8vCPU 16GB RAM!

I wonder if anybody from Fortinet is following these forums or is it just for users to sort out their problems on their own, like open source solutions??

[Carl_Wallmark]

According to support, the fix for this will be included in 1.2.2, however I don´t know when it will be released, but someone said by the end of September. Fingers crossed.....

[CoSax]

For those that are suffering, I increased the keepalive interval to 120 sec and gave me some breathing space... not a lot but is something.

Not sure if there is any negative impact by doing this.