Skip to main content
nmathur
Staff
nmathur
Staff
December 2, 2020

Technical Note: Customers with older community editions facing stoppage of the elasticsearch service

  • December 2, 2020
  • 0 replies
  • 1146 views
Description
In case of customer having older FortiSOAR (CyOPs) community editions the elasticsearch logs fill up rapidly due to the generation of the license expired issue every second. Due to this the /var/log partition disk space gets fills up, which might also cause the elasticsearch service to stop.

To check if this is the issue, you can run the following diagnostic checks:

  1. Check the disk space using the df –h command.
  2. Check the elasticsearch log file using the following command:
    tailf /var/log/elasticsearch/elasticsearch.log 
    In the log you will see the following error (Block1 - Elasticsearch error) repeating every second:

Solution
To resolve this issue, do the following:
  1. Enable the Basic License
  2. Remove Logstash and Kibana
  3. Delete unassigned shards

Enabling the basic license

  1. To stop the Block1 - Elasticsearch error, you need to enable the basic license using the following command: 
    curl --user sa-admin:cybersponse -X POST "https://localhost:9200/_license/start_basic?acknowledge=true" -H 'Content-Type: application/json' 
    Note: You need to use http or https (highlighted in RED) and sa-admin:cybersponse or elastic:<hkey> based on the version of your community edition.   
    You will see the following output:
    {"acknowledged":true,"basic_was_started":true}
  2. Restart the elasticsearch service using the following command: 
    systemctl restart elasticsearch

Removing Logstash and Kibana

  1. To remove logstash, first you must stop the logstash service, then disable the logstash service, and then remove the logstash service, using the following commands:
    systemctl stop logstash
    systemctl disable logstash
    yum remove logstash
  2. To remove kibana, first you must stop the kibana service, then disable the kibana service, and then remove the kibana service, using the following commands:
    systemctl stop kibana
    systemctl disable kibana
    yum remove kibana
  3. Reboot your FortiSOAR (CyOPs) instance.

Deleting Unassigned Shards

  1. Check the elasticsearch.log for the following error:
    /var/lib/elasticsearch/nodes/0/indices/9fksydJQSv-bKLExc6mv7g/4/_state: Too many open files
  2. Delete unassigned shards indices using the following commands: 
    curl --user sa-admin:cybersponse -X GET "https://localhost:9200/_cat/shards | grep UNASSIGNED | awk {'print $1'} | xargs -i curl -XDELETE --user sa-admin:cybersponse "https://localhost:9200/{}"
    curl --user sa-admin:cybersponse -X DELETE "https://localhost:9200/.watcher-*?pretty"
    Note: You need to use http or https (highlighted in RED) and sa-admin:cybersponse or elastic:<hkey> based on the version of your community edition.   
  3. Restart all the services using the following command: 
    csadm services --restart





Terms & ConditionsAccessibility statement