| Solution |
- Requirements: Super require access to update.fortiguard.net on port 443. From node CLI as root, the output should be like:
nmap update.fortiguard.net -p 443
Starting Nmap 7.92 ( https://nmap.org ) at 2024-07-04 14:11 CEST
Nmap scan report for update.fortiguard.net (12.34.97.16)
Host is up (0.094s latency).
Other addresses for update.fortiguard.net (not scanned): 208.184.237.66 173.243.138.71
PORT STATE SERVICE
443/tcp open https
Nmap done: 1 IP address (1 host up) scanned in 0.75 seconds
curl -vk https://update.fortiguard.net
* Rebuilt URL to: https://update.fortiguard.net/
* Trying 12.34.97.16...
* TCP_NODELAY set
* Connected to update.fortiguard.net (12.34.97.16) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Request CERT (13):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Certificate (11):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=California; L=Sunnyvale; O=Fortinet; OU=FDS; CN=fds1.fortinet.com; emailAddress=support@fortinet.com
* start date: May 23 17:18:23 2024 GMT
* expire date: Nov 19 17:18:23 2024 GMT
* issuer: C=US; ST=California; L=Sunnyvale; O=Fortinet; OU=Certificate Authority; CN=support; emailAddress=support@fortinet.com
* SSL certificate verify result: self signed certificate in certificate chain (19), continuing anyway.
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET / HTTP/1.1
> Host: update.fortiguard.net
> User-Agent: curl/7.61.1
> Accept: */*
>
* Empty reply from server
* Connection #0 to host update.fortiguard.net left intact
curl: (52) Empty reply from server
- The collector can resolve and has access to Fortinet repositories: os-pkgs-cdn.fortisiem.fortinet.com, os-pkgs.fortisiem.fortinet.com, os-pkgs-r8.fortisiem.fortinet.com and super on port 443. From collector CLI as root:
curl -vk https://super_ip
* Rebuilt URL to: https://super_ip/
* Trying 10.5.8.91...
* TCP_NODELAY set
* Connected to super_ip (super_ip) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, [no content] (0):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=US; ST=CA; L=SunnyVale; O=Fortinet; CN=localhost
* start date: May 16 09:11:30 2024 GMT
* expire date: May 14 09:11:30 2034 GMT
* issuer: C=US; ST=CA; L=SunnyVale; O=Fortinet; CN=localhost
* SSL certificate verify result: self signed certificate (18), continuing anyway.
* TLSv1.3 (OUT), TLS app data, [no content] (0):
> GET / HTTP/1.1
> Host: super_ip
> User-Agent: curl/7.61.1
> Accept: */*
>
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, [no content] (0):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS app data, [no content] (0):
< HTTP/1.1 200 OK
< Date: Thu, 04 Jul 2024 12:17:17 GMT
< Server: Apache
< X-XSS-Protection: 1; mode=block
< x-Frame-Options: SAMEORIGIN
< X-Content-Type-Options: nosniff
< Strict-Transport-Security: max-age=31536000; includeSubDomains
< Content-Security-Policy: default-src 'self' https://*.duosecurity.com https://*.googleapis.com https://*.gstatic.com; img-src 'self' data: https://maps.googleapis.com https://maps.gstatic.com; script-src 'self' 'unsafe-inline' 'unsafe-eval' https://*.googleapis.com; style-src 'self' 'unsafe-inline' https://*.googleapis.com;
< Referrer-Policy: no-referrer-when-downgrade
< Last-Modified: Tue, 13 Feb 2024 09:08:29 GMT
< ETag: "120-6113fbcdc5711"
< Accept-Ranges: bytes
< Content-Length: 288
< Content-Type: text/html; charset=UTF-8
<
<html>
<head>
<title></title>
<script type="text/javascript">
var hst = location.hostname
document.write("<meta HTTP-EQUIV='REFRESH' content='0; url=https://"+hst+"/phoenix' >")
</script>
</head>
<body>
<b> <h2 align="center"> </h2> </b>
<br><br><br><br><br><br>
</body>
</html>
* Connection #0 to host super_ip left intact
curl -k https://os-pkgs-cdn.fortisiem.fortinet.com
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>301 Moved Permanently</title>
</head><body>
<h1>Moved Permanently</h1>
<p>The document has moved <a href="https://os-pkgs.fortisiem.fortinet.com/centos6/">here</a>.</p>
<hr>
<address>Apache Server at os-pkgs.fortisiem.fortinet.com Port 443</address>
</body></html>
curl -k https://os-pkgs-r8.fortisiem.fortinet.com
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>403 Forbidden</title>
</head><body>
<h1>Forbidden</h1>
<p>You don't have permission to access this resource.</p>
<p>Additionally, a 403 Forbidden
error was encountered while trying to use an ErrorDocument to handle the request.</p>
</body></html>
- If the FortiSIEM is behind a proxy:
- Follow the documentation Upgrade via Proxy.
- Make sure the proxy is configured for update.fortiguard.net from the FortiSIEM GUI at Resource -> Malware IP -> FortiGuard Malware IP -> More -> Update then set the proxy settings and credentials and save:
- Download the update .zip file from the support portal https://support.fortinet.com and verify the checksum. For example from Windows Power Shell:
Get-FileHash FSM_Upgrade_All_7.2.1_build0241.zip -Algorithm MD5 | Format-List
The checksum should match the hash provided in the support.fortinet.com. If not, try to re-download the .zip file.
- Check collector and super Health: Go to Admin -> Health, make sure super and collector are shown green and in the normal state, and check the currently running version. If not, do not proceed with the upgrade and fix the super and collector first. Related article: How to troubleshoot collector issues.
- Load image on the super: Clean for previously loaded images under Admin -> Settings -> Image Server, remove the previously loaded images, and clear the Version field:
- Make sure that no file remains in the next directories on super CLI as root:
ls -l /opt/phoenix/CollectorUpgrade/
total 0
ls -l /opt/phoenix/cache/installedimages/collector/
total 0
- Configure the Custom Update section with an IP or Fully Qualified Domain Name that the collector can reach to super on port 443 without filtering and Save.
Important:
This can differ along with all the running collectors and how their resolving and registration configuration has been set. It means this step may require to be renewed from here for each collector.
- In the Collector section, enter the Version (X.X.X) to upload Select the file from the local machine, and select Upload File:
If having a Checksum error, make sure the Version field is correctly set along with the Image version uploaded and review the proxy settings as the update.fortiguard.net is accessed to do this check.
- If necessary, check for explicit errors in the logs:
tail -50f /opt/glassfish/domains/domain1/logs/phoenix.log
2024-07-08 10:48:44,287 [http-listener-2(20)] INFO com.ph.phoenix.framework.logging.PhAudit - [PH_AUDIT_OBJECT_CREATED]:[phCustId]=1,[eventSeverity]=PHL_INFO,[phEventCategory]=2,[procName]=AppServer,[user]=admin(su),[srcIpAddr]=172.26.128.202,[osObjType]=System configuration,[osObjName]=collector_image_name,[phLogDetail]=System configuration has been created
2024-07-08 10:48:44,288 [http-listener-2(20)] INFO com.ph.phoenix.framework.logging.PhAudit - [PH_AUDIT_OBJECT_CREATED]:[phCustId]=1,[eventSeverity]=PHL_INFO,[phEventCategory]=2,[procName]=AppServer,[user]=admin(su),[srcIpAddr]=172.26.128.202,[osObjType]=System configuration,[osObjName]=collector_image_version,[phLogDetail]=System configuration has been created
2024-07-08 10:48:44,293 [http-listener-2(20)] INFO com.ph.phoenix.framework.logging.PhAudit - [PH_AUDIT_OBJECT_CREATED]:[phCustId]=1,[eventSeverity]=PHL_INFO,[phEventCategory]=2,[procName]=AppServer,[user]=admin(su),[srcIpAddr]=172.26.128.202,[osObjType]=System configuration,[osObjName]=Collector_Download_Url,[phLogDetail]=System configuration has been created
2024-07-08 10:49:15,947 [http-listener-2(20)] INFO com.ph.phoenix.framework.logging.PhAudit - [PH_AUDIT_OBJECT_CREATED]:[phCustId]=1,[eventSeverity]=PHL_INFO,[phEventCategory]=2,[procName]=AppServer,[user]=admin(su),[srcIpAddr]=172.26.128.202,[osObjType]=System configuration,[osObjName]=COLLECTOR_DOWNLOAD_PASSWORD,[phLogDetail]=System configuration has been created
2024-07-08 10:49:15,973 [http-listener-2(20)] INFO com.ph.phoenix.framework.logging.PhAudit - [PH_AUDIT_UPLOAD_COLLECTOR_AGENT_IMAGE]:[phCustId]=1,[eventSeverity]=PHL_INFO,[phEventCategory]=2,[procName]=AppServer,[user]=admin,[srcIpAddr]=172.26.128.202,[files]=FSM_Upgrade_All_7.2.1_build0241.zip,[version]=7.2.1,[phLogDetail]=Upload collector installation image
- Check the generated files from super CLI as root:
find /opt/phoenix/CollectorUpgrade/ -type f -ls
209100088 2901392 -rw-rw-r-- 1 root admin 2971024346 Jul 8 10:49 /opt/phoenix/CollectorUpgrade/FSM_Upgrade_All_7.2.1_build0241.zip
4625025 2901292 -rwxr-xr-x 1 root admin 2970921023 Jul 8 10:52 /opt/phoenix/CollectorUpgrade/tar/FSM_Upgrade_All_7.2.1_build0241.zip
Take note of the .zip file size for the next steps.
- Check the generated URL for the collector to download the image:
psql -U phoenix phoenixdb -c "select property,value from ph_sys_conf where property ilike 'collector_image%' or property ilike '%Download_Url';"
property | value
-------------------------+-----------------------------------------------------------------------------------------------------------
collector_image_name | FSM_Upgrade_All_7.2.1_build0241.zip
collector_image_version | 7.2.1
Collector_Download_Url | https://fsm721ClickHSup:443/CollectorUpgrade/FSM_Upgrade_All_7.2.1_build0241.zip
Image_Download_Url | https://Collector:gepPIzjNgP@fsm721ClickHSup:443/CollectorUpgrade/tar/FSM_Upgrade_All_7.2.1_build024...
(4 rows)
Take note of the Image_Download_Url for the next step.
- Download image: From the collector CLI, check that the collector can access the generated URL (without the /tar directory):
curl --head -ik https://Collector:gepPIzjNgP@fsm721ClickHSup:443/CollectorUpgrade/FSM_Upgrade_All_7.2.1_build0241.zi...
HTTP/1.1 200 OK
Date: Mon, 08 Jul 2024 14:52:51 GMT
Server: Apache
X-XSS-Protection: 1; mode=block
x-Frame-Options: SAMEORIGIN
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubDomains
Content-Security-Policy: default-src 'self' https://*.duosecurity.com https://*.googleapis.com https://*.gstatic.com; img-src 'self' data: https://maps.googleapis.com https://maps.gstatic.com; script-src 'self' 'unsafe-inline' 'unsafe-eval' https://*.googleapis.com; style-src 'self' 'unsafe-inline' https://*.googleapis.com;
Referrer-Policy: no-referrer-when-downgrade
Last-Modified: Mon, 08 Jul 2024 14:11:27 GMT
ETag: "b1163bda-61cbcfd2f6c91"
Accept-Ranges: bytes
Content-Length: 2971024346
Content-Type: text/html; charset=UTF-8
Make sure that the HTTP code is 200 and Content-Length matches the previously noted size. If the result is not similar, check the network settings again or the state of the super.
- Clean the destination directory on the collector CLI as root:
cd /opt/upgrade
rm -rf *
- Clean the previous running download actions using the collector ID from Admin -> Health -> Collector Health or the next commands from super CLI as root:
psql -U phoenix phoenixdb -c "select col.collector_id,col.ip_addr,org.name,col.name from ph_sys_collector as col inner join ph_sys_domain as org on col.cust_org_id=org.domain_id;"
collector_id=10001 <- Update with the appropriate collector ID.
psql -U phoenix phoenixdb -c "delete from ph_task where type='DownloadImage' and collector_id=$collector_id"
- Under Admin -> Health -> Collector Health, select the collector -> Actions, and select 'Download Image':
- Check for explicit errors in the logs on both nodes:
tail -f /opt/phoenix/log/phoenix.log /opt/glassfish/domains/domain1/logs/phoenix.log
- Check the status under Admin -> Health -> Collector Health -> Display Columns Tick 'Download Status' and see the result.
- Check the final file on the collector:
ls -l /opt/upgrade/
total 2901392
-rw-rw-r-- 1 root admin 2971024346 Jul 8 16:21 FSM_Upgrade_All_7.2.1_build0241.zip
md5sum /opt/upgrade/FSM_Upgrade_All_7.2.1_build0241.zip
c552f8d89f6dadf68d10a3e97279bae2 /opt/upgrade/FSM_Upgrade_All_7.2.1_build0241.zip
- If the download fails, make sure the connection is stable, check for explicit errors, and retry this 'Download Image' section.
- Optional: Run the download image manually from the collector CLI as root (if the download image is done manually, installation of the image will have to be done manually too):
mkdir /opt/upgrade
chmod 775 /opt/upgrade/
wget --no-check-certificate https://Collector:gepPIzjNgP@fsm721ClickHSup:443/CollectorUpgrade/FSM_Upgrade_All_7.2.1_build0241.zi... -P /opt/upgrade/
--2024-07-25 17:32:18-- https://Collector:*password*@fsm721ClickHSup:443/CollectorUpgrade/FSM_Upgrade_All_7.2.1_build0241.zip
Connecting to 10.5.8.40:443... connected.
WARNING: The certificate of ‘fsm721ClickHSup’ is not trusted.
WARNING: The certificate of ‘fsm721ClickHSup’ hasn't got a known issuer.
The certificate's owner does not match hostname ‘fsm721ClickHSup’
HTTP request sent, awaiting response... 401 Unauthorized
Authentication selected: Basic realm="Collector Upgrade"
Reusing existing connection to fsm721ClickHSup:443.
HTTP request sent, awaiting response... 200 OK
Length: 2971024346 (2.8G) [text/html]
Saving to: ‘/opt/upgrade/FSM_Upgrade_All_7.2.1_build0241.zip’
FSM_Upgrade_All_7.2.1_build0241.zip 100%[===========================================================================>] 2.77G 95.5MB/s in 31s
2024-07-25 17:32:50 (91.1 MB/s) - ‘/opt/upgrade/FSM_Upgrade_All_7.2.1_build0241.zip’ saved [2971024346/2971024346]
chown -R root:admin /opt/upgrade/
chmod 664 /opt/upgrade/FSM_Upgrade_All_7.2.1_build0241.zip
md5sum /opt/upgrade/FSM_Upgrade_All_7.2.1_build0241.zip
c552f8d89f6dadf68d10a3e97279bae2 /opt/upgrade/FSM_Upgrade_All_7.2.1_build0241.zip
- Install image: Clean the previous Upgrade image state on the super from super CLI as root:
psql -U phoenix phoenixdb -c "select col.collector_id,col.ip_addr,org.name,col.name from ph_sys_collector as col inner join ph_sys_domain as org on col.cust_org_id=org.domain_id;"
collector_id=10001 <- Update with the appropriate collector ID.
psql -U phoenix phoenixdb -c "delete from ph_task where type='UpgradeImage' and collector_id=$collector_id"
- Check basic. From the collector CLI as root, run the following:
python --version
python3 --version
yum clean all
yum check-update
Make sure to have no error while executing those commands and obsolete package list is displayed.
- Make sure python3.9 packages are installed:
pip3.9 list | wc -l
181
If 0 packages are listed, the collector upgrades to v7.X may fail with 256 error code and "ModuleNotFoundError: No module named 'psycopg2'" error. In that case, install missing packages using the steps described in: How to fix 256 error during upgrade .
- Install 7zip tool if missing from collector CLI as root:
yum install -y p7zip
- Clean for previous uncompressed directory:
cd /opt/upgrade/
ls -d */
FSM_Upgrade_All_7.2.1_build0241/
rm -rf FSM_Upgrade_All_7.2.1_build0241
- Take a collector VM snapshot. If running on hardware, the upgrade process will perform a backup before proceeding.
- Go to Admin -> Health -> Health Collector, select 'Collector' -> Action, and select 'Install Image'.
- Check the progress in the collector logs:
tail -f /opt/phoenix/log/collector-upgrade.log
- Wait for the automatic reboot.
- Optional: Run the installation manually, from collector CLI as root:
screen -S upgrade
/usr/bin/python /opt/phoenix/phscripts/bin/phcollectorimageinstaller.py FortiSIEM |tee -a /opt/phoenix/log/collector-upgrade.log
[PH_MODULE_LOCAL_CONFIG_LOADED]:[eventSeverity]=PHL_INFO,[procName]=phtools,[fileName]=phConfigLoader.cpp,[lineNumber]=168,[configName]=global,[phLogDetail]=Module loaded local config successfully
....
[18:42:18] cleanup : CLEANUP | Remove data-definition SVN config xml ...| localhost | SUCCESS | 268ms
[18:42:19] cleanup : CLEANUP | Remove some temp files no longer needed ...| localhost | CHANGED | 1.83s
Operation upgrade Failed with the error above
/bin/cp: cannot stat '/opt/phoenix/config/sys/etc/mod_evthandler.so': No such file or directory
/usr/local/bin/configureFSM.py -o upgrade -z Europe/Paris -r collector -u 7.2.1.0241
configureFSM.py returns 0
reboot
- Check the final status: Under Admin -> Health -> Collector Health, Display Columns, tick Version, Install Status, validate the new version, the state, and the last file received:
- From the collector CLI as root, phstatus command should show all the processes running.
- In case of error: Check for explicit errors from collector CLI as root:
- In the main upgrade log file: /usr/local/upgrade/logs/ansible.log.
- In /opt/phoenix/log/collector-upgrade.log.
- If the issue is not explicit, run phziplogs /tmp/failed_upgrade 1 and provide the generated file and the result of all previous steps to the support team.
- If the error is not explicit and the collector requires to be up urgently, take the logs and revert to the previous VM snapshot or restore the upgrade there Upgrading to FortiSIEM 7.2.x
- If the upgrade went well but health is not normal, review How to troubleshoot collector issues
- Try to upgrade with manual steps if all previous required checks have passed.
|
