Maintaining many collector versions from the All in One for many clients as a MSP

[KarlH]

We have Collectors  at 6.2, 6.3, 6.5.0,  6.5.1 , 6.6, 6.7,  7.1 , 7.0,  7.1 and 7.2

Supervisor is 7.1.3

I have taken on a new role out of the versions 6.5 collector seems to a problem, as a Engineer I am the one who has to answer collectors going in the red for disk space capacity, cpu usage, and what ever else.

And now I'm told the supervisor image server is broken  7.1.3.  How do we get that fixed?

Do I try to upgrade 6.5 ? can I see the upgradable paths somewhere for the above versions how do we get the supervisor image server fixed so it can be used and function correctly for them we need to improve our KPI.

Meanwhile the worst is that these clients repeatedly lose the credentials to LOG IN  to the collector, not register,  and this forces us to have to re-deploy the collector,  what version do I give them that is the least problematice I haven't done that yet either... need some best practice advice and assist. All in one single collectors not cluster, need upgrade paths.

Thank you in advance

[premchanderr]

HI @KarlH ,

Every FortiSIEM version has lot of fine tuning and enhancement to ensure SIEM works in optimal efficiency. It is strongly advisable to have collectors in same version or one lesser version than supervisor. Collector on 6.5 and Supervisor on 7.1.3 the gap is long, many enhancements would be missing. 

I don't understand supervisor image server broken, this doesn't happen. Do you have any bug ID on this or do provide further information.

Collectors are deployed on Rocky Linux OS, you can follow linux forums on steps to reset root account instead of re-deploying it if customers lose their credentials.  

[KarlH]

Hi Prem, @premchanderr 

The procedures are different for different scenarios what am I looking for 'How to reset a password on Rocky Linux on a VM on AZure cloud?" which is what they have.  I  have never done it before. We get customers always saying they lost the creds to login into Rocky Linux VM.

What version of Rockey Linux does collector 7.1 run on?

Have  you seen this? https://forums.rockylinux.org/t/how-to-recover-root-password/3433/8

and this ? https://forums.rockylinux.org/t/i-forgot-my-root-password-but-traditional-single-user-mode-doesnt-wo...

There are issues with doing that   Does Fortinet have hard and fast safe set of procedures for doing this with a customer?

Thank you

[KarlH]

Hi @premchanderr Prem, How can I find the release notes and bug fixes for Collector version 7.1.3? So I can build a case for stopping the installing of 6.5 that people are doing?

[KarlH]

@premchanderrHI Prem When you say the image sever doesn't get broken what do you mean  how can I get it working to up update collectors running 6.5 to ar newer version? 

appreciate your help.  I don't have any info the people who maintained it left how can I gather info for you?

[premchanderr]

hi @KarlH ,

You would need to simply upload the FSM Upgrade zip file to Image server.

Documentation (Upgrade Collectors):
https://docs.fortinet.com/document/fortisiem/7.2.3/upgrade-guide/505373#Installa4

If you face any issue then its specific to your environment, better to open ticket with Fortinet Support.

[premchanderr]

Release notes:
https://docs.fortinet.com/document/fortisiem/7.1.5/release-notes/46171/whats-new-in-7-1-5

[KarlH · Accepted solution]

@Prem

So the way the image server can be "broken" is due to network or even more obscure issues

only by placing things in debug

su - admin
phtools --change-log ALL DEBUG
exit

Disable modsecurity

mv /etc/httpd/conf.d/mod_security.conf /tmp/

- Reload http service

# systemctl reload httpd

were we able to successfully upload a zip file,  Second I am told we must hit the upload button a second time

what does this do there is not documentation for this gui functionality under

ADMIN -> SETTINGS -  IMAGE SERVER

to help know how this feature works or what is supposed to happen I would be grateful for a clear step by step process for using this feature and knowing what is expected to occur during the process.

After getting the first step completed so the  .zip file exists as expected in the

/opt/phoenix/CollectorUpgrade/    

But the ImageS etup task was not completed:

https://community.fortinet.com/t5/FortiSIEM/Technical-Tip-How-to-upgrade-collectors-from-the-FortiSI...

the doc says to run

psql phoenixdb phoenix -c "SELECT type, progress from ph_task where type = 'ImageSetup';"

the result for me was

type | progress
------------+----------
ImageSetup | 0                    What do I need to do to get the ImageSetup task to complete?

(1 row)

/opt/phoenix/CollectorUpgrade/   has a file in it in red--- FSM_Upgrade_All_7.1.3_build0165.zip

Finally does moving away from 7.1.3 to a much more recent version make this process more robust and less buggy, as I am told by support this feature is problematic, but it is a crucial tool for MSSP  trying to support many clients with many versions of collector and in keeping with best practices of mantinaing equitable versions of supe4rvisor and collector.

Thank you

thank you in advance

[Secusaurus]

Hi @KarlH,

The output you have here is the one to expect right after uploading and should change after a few minutes to 100 or disappear completely (as it is the image prep task you see here).

As far as I understand, it is recommended to do every update on the cluster also on the collector nodes afterwards - which should be easy using the mentioned image upload feature.

As the collectors have access and credentials to vital components in the customers' environments, applying patches to them is a critical requirement. So, I would propose to resolve this variety of versions in near future.

In newer versions of FSM (7.2.1?), you have the option to create a HA-cluster for collectors, in case upgrading was an issue because of losing logs in that time frame...

And one final word about updates: Yes, it is always something I fear. But we never had issues with collectors, only with supervisor/worker nodes. So, upgrading collectors is usually a smooth process.

Best,

Christian

[premchanderr]

Hi Karl,

You can use the documentation only if you face any issue, otherwise following the upgrade file keeps it simple.

After uploading zip file and if this can be viewied in Image server GUI, then go ahead to "Download Image" and "Install Image" on the collector.

If still issue persists then some problem or misconfig on your environment, better to open in Fortinet Support portal.