Skip to main content
KarlH
KarlH
Explorer II
October 14, 2024
Solved

Disableing Clickhouse clickhouse-server.service,

  • October 14, 2024
  • 3 replies
  • 6042 views

What are the ramifications of disabling the clickhouse-server.service,

Or asked another way what criteria would I need in order to make an informed decision? 

Id like a little more knowledge about clickhouse and if its set up by default and how to know if it actually is being used.I I don't think the log rotation is set up on a lot of the clients collectors at any rate so periodic removal of logs has to take place until there is a final solution.

 

Thank you in advance.

 

Best answer by premchanderr

Hi @KarlH ,

 

To disable clickhouse on collector:

systemctl stop clickhouse-server.service
systemctl disable clickhouse-server.service

 

Detailed Documentation: https://community.fortinet.com/t5/FortiSIEM/Troubleshooting-Tip-High-root-disk-usage-on-collector-due-to/ta-p/243281

Note: These steps are only for collector, do not run on super or worker.

3 replies

premchanderr
Staff & Editor
premchanderr
Staff & Editor
October 15, 2024

Hi @KarlH ,

 

Collectors don't require clickhouse service.  You could follow the below document to resolve this issue:

 

https://community.fortinet.com/t5/FortiSIEM/Troubleshooting-Tip-High-root-disk-usage-on-collector-due-to/ta-p/243281

Secusaurus
Secusaurus
Contributor III
October 15, 2024

Hi @KarlH,

 

Clickhouse is a database application that can (but does not need to be) used with FortiSIEM. You can google for clickhouse to learn more about it, it is not related to FSM directly.

That being said, FSM deployment strongly recommends using Clickhouse, because it is tested, and also integrated in the GUI. Clickhouse needs a "controlling" instance ("Keeper") that can run on a supervisor (or workers) and the database instances ("Replicas" in "Shards") that are installed with workers (and supervisor). You would usually set up multiple workers to serve the Clickhouse database. So, in conclusion, clickhouse should run on the supervisor and the workers (in case you use that and not NFS, Elastic,..).

You can read more in the reference design docs and the Fortinet FCP training for FSM. 

 

Best,

Christian

NSE8 | Fortinet Advanced MSSP Partner
KarlH
KarlHAuthor
Explorer II
October 15, 2024

Ok thank you Christian and Prem!

I was looking into how to deploy a collector and coincidentally I discovered there is an option during install from the 6.5 all in one install docs., specifically installing a new collector, where you can specify the type of  EDB (Event Data Base). 

 

 I was told we were using the NFS EDB.

 

The big question for me is how can I check all this on the super/collector and workers?  and how do collectors / workers and supers work together if the super and workers use NFS but a collector has clickhouse? ( meanwhile the clickhouse logs max out the disk space under /var on SOME clients collectors and not others( which we then have to run the rm -rf  command),

Finally  Prem said collectors don't require clickhouse, or any EDB. What are the perquisites where they do? or don't?  Can I find my answers in the GUI?or query the client collector for configuration settings?  

!Screenshot 2024-10-15 081757.png

 

Thanks again

Secusaurus
Secusaurus
Contributor III
October 16, 2024

Hello @KarlH,

 

I think there is some knowledge missing concerning the database types.

As you can see in the screenshot, FSM supports different kinds of databases. EventDB (local, or on NFS), ClickHouse and Elasticsearch.

This database will be used for all the events (think of it like a SQL database, although SQL would not be the right one to serve here). Logs are sent or received through collectors and sent upstream to the workers (or, if no workers exist, to the supervisor). They then get parsed and stored in a database via the workers.

In a simple setup, this must be a single database to where every cluster member has access to. Obviously, an EventDB on local disk cannot be available for other members. So, for scaling reasons you'd prefer NFS, or (better!) ClickHouse.

The cluster members that have read or write access to the database need to have their daemons running to access the specific type of database. So, if you deployed ClickHouse, the workers and supervisor will usually run the ClickHouse-services.

In your case, as you do not use ClickHouse at all, there should not run any ClickHouse-related process, or, at least, not consume resources.

 

Have a look at the reference design: https://docs.fortinet.com/document/fortisiem/7.2.3/fortisiem-reference-architecture-using-clickhouse/105365/what-is-the-fortisiem-reference-architecture

 

And I strongly recommend the Fortinet Training for that one, to learn more about how the cluster works: https://training.fortinet.com/local/staticpage/view.php?page=library_fortisiem

 

Best,

Christian

NSE8 | Fortinet Advanced MSSP Partner
    KarlH
    KarlHAuthor
    Explorer II
    October 16, 2024

    HI Christian we  send our customers instructions for the All in One installation portion of the pdf. Its only for them to install a single collector possibly more in a hypervisor environment.  I have not yet done a dry run of that installation to see what the customer sees. I have been told the collectors are not using clickhouse, whatever the workers and and  supervisor use is another matter,  A small portion of our customers collectors are getting filled with the log files of clickhouse, it is causing problems.  1) Some customers are not savvy Linux users and they don't know to run df -h or rm -rf commands, nor do they even check, they consider it a set it and forget scenario 2) the other problem is that they loose the credentials to the collector both of these require us as engineers to repeatedly contact them to log on, assuming they have the creds and clean up the disk.  I'm concerned that 1) clickhouse should even be running on the collector and 2 if the instructions should be forgoing that option for this simple scenario, because they wont know what to select  3) if the credential are lost we have to redeploy another collector, again giving them the installation instruction written by Fortinet.  I am trying to mitigate these maintenance issues.

    Terms & ConditionsAccessibility statement