A FortiClient EMS HA installation described in EMS nodes in HA with Postgres DB HA cluster using EMS PostgreSQL HA Docker, single region uses Docker volumes which get created on the fly with the sudo docker container run command when spinning up a DB container (whether DB or witness node). An issue where the container gets stuck in the restarting state may be observed by executing the sudo docker container ls command:
The following is a command excerpt based on the administration guide:
docker run --restart always --detach --name pg-1 -p 5432:5432 \\
--env REPMGR_PRIMARY_HOST=pg-1 \
--env REPMGR_PRIMARY_PORT=5432 \
--env REPMGR_PARTNER_NODES=pg-1,pg-2,pgw-1:5432 \
--env REPMGR_NODE_NAME=pg-1 \
--env REPMGR_NODE_NETWORK_NAME=pg-1 \
--env REPMGR_PORT_NUMBER=5432 \
--volume pg_1_data:/fortinet/postgresql \
--volume <path to conf.d>:/fortinet/postgresql/conf/conf.d \
Note the --volume variables.
Here, pg_1_data is a volume’s name and /fortinet/postgresql is container’s volume path. When docker container run command is used, Docker creates the volume (empty) and mounts it at /fortinet/postgresql, where then container writes data to (in this case PostgreSQL-related data).
Docker volumes are persistent in nature and independent of a container lifecycle. If there is something wrong with the DB container, it can be deleted. However, the data will still be there in a volume. When creating a new container with the docker container run command, it can be attached to that volume to have the same data as the older one.
This is where the issue may occur. Due to the complexity of the setup, it is common to run the docker startup command multiple times to fix minor issues like typos or adjust some of the variables. However, since the volume has already been created on the first execution of the docker container run command and is simply re-used thereafter, the original volume holding outdated and incorrect configuration applies a wrong PostgreSQL (repmgr) configuration, causing the container to fail to initiate.
One of the possible outcomes is shown on the screenshot below:
To view logs, like in the above screenshot, run:
sudo docker container log <container id>
To address this, before every new execution of docker container run command that is meant to address misconfigurations and mistakes in the environment variables and command parameters, remove the old volume first:
sudo docker volume ls
sudo docker volume rm <volume name>
Related article:
Troubleshooting Tip: FortiClientEMS PostgreSQL docker container node keeps restarting | 
