EMS 1.2.1: Machine hangs very often

[CoSax]

Our EMS is now version 1.2.1. Version 1.0.0 was initially installed and then upgraded to the newer version whenever it was available.

The problem we face now is quite severe. The machine hangs at least once a day and the only thing you can do is restart. I increased considerably both memory and CPU but no improvement. The EMS logs indicate the following errors:

[08-03 07:18:27][  ERROR]: [0x1aecb768] error:00000005:lib(0):func(0):DH lib [08-03 07:18:27][  ERROR]: [0x1aecb768] fatal_error_occurred, session_close [08-03 07:22:54][  ERROR]: [0x1ad969c0] BIO_pending fatal_error_occurred = true; 5 [08-03 07:22:54][  ERROR]: [0x1ad969c0] error:00000005:lib(0):func(0):DH lib [08-03 07:22:54][  ERROR]: [0x1ad969c0] fatal_error_occurred, session_close [08-03 07:24:42][  ERROR]: [0x1ada6c08] fatal_error_occurred, session_close [08-03 07:25:02][  ERROR]: [0x1b2466b8] fatal_error_occurred, session_close [08-03 07:25:28][  ERROR]: [0x1ac9ffd0] fatal_error_occurred, session_close [08-03 07:25:50][  ERROR]: [0x1ac9ffd0] fatal_error_occurred, session_close [08-03 07:26:00][  ERROR]: [0x1b19e1f8] fatal_error_occurred, session_close [08-03 07:26:06][  ERROR]: [0x1ae45108] fatal_error_occurred, session_close [08-03 07:26:12][  ERROR]: [0x1b2466b8] fatal_error_occurred, session_close [08-03 07:27:30][  ERROR]: [0x1b2df3a0] fatal_error_occurred, session_close [08-03 07:28:26][  ERROR]: [0x1af42de8] fatal_error_occurred, session_close [08-03 07:30:13][  ERROR]: [0x1af62d38] Error : 997, session_accept renew_session status : 1 [08-03 07:30:30][  ERROR]: [0x1ad969c0] fatal_error_occurred, session_close [08-03 07:30:57][  ERROR]: [0x1b26eb88] fatal_error_occurred, session_close [08-03 07:31:24][  ERROR]: [0x1ada6c08] fatal_error_occurred, session_close [08-03 07:31:35][  ERROR]: [0x1b1ae200] Error : 997, session_accept renew_session status : 1 [08-03 07:31:36][  ERROR]: [0x1b1ae200] fatal_error_occurred, session_close [08-03 07:31:37][  ERROR]: [0x1b2ef5e8] fatal_error_occurred, session_close [08-03 07:31:56][  ERROR]: [0x1af62d38] Error : 997, session_accept renew_session status : 1

In the event log the following errors appears quite often:

Log Name:      Application Source:        Application Error Date:          3/8/2017 7:32:02 πμ Event ID:      1000 Task Category: (100) Level:         Error Keywords:      Classic User:          N/A Computer:      Description: Faulting application name: FcmDaemon.exe, version: 1.2.1.394, time stamp: 0x595ebc77 Faulting module name: policyhelper.dll, version: 1.2.1.394, time stamp: 0x595ebc63 Exception code: 0xc0000005 Fault offset: 0x00021300 Faulting process id: 0x8f4 Faulting application start time: 0x01d30b794f4c3799 Faulting application path: C:\Program Files (x86)\Fortinet\FortiClientEMS\FcmDaemon.exe Faulting module path: C:\Program Files (x86)\Fortinet\FortiClientEMS\policyhelper.dll Report Id: b201e0ee-7804-11e7-8107-005056840414 Faulting package full name: Faulting package-relative application ID:

Log Name:      System Source:        Service Control Manager Date:          3/8/2017 7:32:26 πμ Event ID:      7031 Task Category: None Level:         Error Keywords:      Classic User:          N/A Computer:      Description: The FortiClient Enterprise Management Server service terminated unexpectedly.  It has done this 255 time(s).  The following corrective action will be taken in 2000 milliseconds: Restart the service.

Any ideas how to sort this out??

[BenneP]

Hi,

We currently are experiencing similar issues with FortiClientEMS 1.2.1.0394.

After about 24h of uptime, the Memory usage on the server tops at 100% and we see multiple MSSQL errors in the Windows applications logs, and the same event you mentionned in the system logs, the Service terminated unexpectedly. Restarting services does not help, it needs a reboot to clear memory.

Here is the MSSQL error:

Log Name:      Application Source:        MSSQL$FCEMS Date:          8/3/2017 8:30:01 AM Event ID:      8645 Task Category: Server Level:         Error Keywords:      Classic User:          SYSTEM Computer:      servername.domain.local Description: A timeout occurred while waiting for memory resources to execute the query in resource pool 'internal' (1). Rerun the query.

Running on Windows Server 2012 R2, in a VMware ESXi 6.5 environment, 8GB RAM, was working fine before updating to 1.2.1.

I'll keep you updated if I find anything.

[Arne_Vanderheyden]

I've got pretty much the same at a client, who recently upgraded to 1.2.1 hoping to get rid of all the 1.0.x misery. Did you already open a ticket for this or not yet?

[Carl_Wallmark]

Same issue here, crashes all the time....

Faulting application name: FcmDaemon.exe, version: 1.2.1.394, time stamp: 0x595ebc77 Faulting module name: policyhelper.dll, version: 1.2.1.394, time stamp: 0x595ebc63 Exception code: 0xc0000005 Fault offset: 0x00021300 Faulting process id: 0xd70 Faulting application start time: 0x01d31107c00ea3ad Faulting application path: C:\Program Files (x86)\Fortinet\FortiClientEMS\FcmDaemon.exe Faulting module path: C:\Program Files (x86)\Fortinet\FortiClientEMS\policyhelper.dll Report Id: 0f021ccc-7cfb-11e7-80d6-005056b31870 Faulting package full name: Faulting package-relative application ID:

[BenneP]

So far so good, I did a snapshot and a full backup beforehand, then installed SQL server express 2014 SP2 (was SP1), and it didn't crash in about a week. I thought it was worth a shot.

I haven't open a ticket myself.

[jklems]

We have had similar issues. Runs fine for about a week, then we have to reboot it.  Our EMS was installed on a fresh HyperV guest , running Windows 2012 Standard, with 1 Virtual socket , 4 virtual processors and 12GB of startup RAM.  I know this seems ridiculous; but when I only assign, 2 virtual CPUs, and 8GB of RAM, it seems to hang more.  While we realize SQL is going to eat up any amount of RAM we throw at it, we suspect some kind of odd SQL memory leak issue.  Were just not sure, and HyperV resource manager is sort of useless in this exercise.  We gauge performance based on how snappy the EMS web admin portal is.  

[jklems]

Well, it happened again.  Almost like clockwork.  works great for about a week and then all of the sudden we can log into the web interface, or the web interface becomes so slow , its unusable.  We reboot it and see how it goes.

[Carl_Wallmark]

Anyone found a solution or workaround ?

EMS is totally useless with 1.2.1... 

[CoSax]

I have opened a ticket but support response times are very disappointing. I understand they plan an update with a fix by the end of the month. I have this problem more than 2-3 months now (with the previous version as well) and having to wait for so long with such an unusable management software is unacceptable.

[Carl_Wallmark]

I also have a ticket but they have not provided anything useful.

I have narrowed it down to when clients tries to connect/upload data. I changed the IP of the server to something else then it was stable as no one could connect to the server, as soon as I change back and clients starts to connect the service crashes over and over and over and again.....