Troubleshooting Tip: How to clear application server cache

When the FortiSIEM is not providing access to the GUI and/or ph services are down after a rough power outage or an upgrade, clearing the application server cache can be useful to start the application server.

Note: Do not clear the cache on Super in HA/DR.

Here are the steps to follow:

From the supervisor CLI as root:

su admin

phtools --stop ALL

monctl stop
cd /opt/phoenix/cache/
mv ip_dir ipdir_.old # replace the ip_dir by the IP of the super
cd /opt/glassfish/domains/domain1/
rm -rf generated/
rm -rf osgi-cache/
kill -9 $(cat /opt/glassfish/domains/domain1/config/pid)

monctl start

watch -n5 phLicenseTool --showDatabasePassword

Wait for the database password to be displayed. This can take a while, along with the resources and the size of the CMDB (around 15 minutes).

Once the password is displayed, press 'Ctrl+C' to stop, which means that the application server is running and the GUI can be accessed.

Check for other services to come up with the following command:

phstatus

Health can also be checked in the GUI under Admin -> Health -> Cloud Health. The status should become 'normal' after a while.

If the application service does not remain accessible, further analysis is required by the support team by sending a log as described in the following article: Technical Tip: How to retrieve logs from FortiSIEM VA and deliver them to support.

Note about clearing cache:

Do not clear the full content of the cache entries (rm -rf /opt/phoenix.cache/*:(

Unlike mv ip_dir ipdir_.old (replace 'ip_dir' with the IP of the super), which only renames one specific cache folder, this will:

• Removes ALL cached runtime data.

• Deletes:

  • Rule engine state.

  • Session data.

  • Cluster sync info.

  • Possibly temporary DB sync files.

Consequences:

• Can cause cluster desynchronization.

• May force a full resync from master.

• Can trigger huge CPU spikes.

• Can break the follower node startup.

• Sometimes causes data loss in rule processing queues.

This is why the best practice is to rename only the problematic folder, never wipe everything.