Troubleshooting Tip: Disaster Recovery EMS Linux Log Collection

[MZBZ]

Description	This article how to collect logs when contacting Fortinet TAC in case an EMS Linux server is not working as expected.
Scope	Linux-based FortiClient EMS v7.4.x.
Solution	If a Linux-based EMS v7.4.x server has stopped responding and is not working as expected, run the following one-line Linux command and attach the created all_logs.tar.gz file to the TAC ticket for further investigation. Since copying and pasting can add additional or hidden characters, the command is also attached to this article as a text file (ems_linux_oneliner_log_collector_v6.zip). This command collects some important log files and a general system status required for troubleshooting issues by the TAC team. It is important to note that this is a single long command and that all of the lines should therefore be selected, copied, and pasted in one run. mkdir ./FortinetSupport && cd ./FortinetSupport && \ echo -e "\n\n###Output of sudo date:\n" > ./FortinetSupport.log ; sudo date >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo uptime:\n" >> ./FortinetSupport.log ; sudo uptime >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo mount:\n" >> ./FortinetSupport.log ; sudo mount >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo findmnt:\n" >> ./FortinetSupport.log ; sudo findmnt >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo cat /etc/fstab:\n" >> ./FortinetSupport.log ; sudo cat /etc/fstab >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo df -Th:\n" >> ./FortinetSupport.log ; sudo df -Th >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo hostnamectl:\n" >> ./FortinetSupport.log ; sudo hostnamectl >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo uname --all:\n" >> ./FortinetSupport.log ; sudo uname --all >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo cat /etc/X11/default-display-manager:\n" >> ./FortinetSupport.log ; sudo cat /etc/X11/default-display-manager &>> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo du / -a -d 1 -h 2> /dev/null\n" >> ./FortinetSupport.log ; sudo du / -a -d 1 -h 2> /dev/null >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo du /opt/ -a -d 2 -h 2> /dev/null\n" >> ./FortinetSupport.log ; sudo du /opt/ -a -d 2 -h 2> /dev/null >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo du /var/lib/postgresql/ -a -d 4 -h 2> /dev/null\n" >> ./FortinetSupport.log ; sudo du /var/lib/postgresql/ -a -d 4 -h 2> /dev/null >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo lsblk:\n" >> ./FortinetSupport.log ; sudo lsblk >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo systemctl --all --type=service | grep -iE 'fort|fcems|apache|redis|postgres'\n" >> ./FortinetSupport.log ; sudo systemctl --all --type=service | grep -iE 'fort|fcems|apache|redis|postgres' >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo lspci:\n" >> ./FortinetSupport.log ; sudo lspci -kvv >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo lshw:\n" >> ./FortinetSupport.log ; sudo lshw >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo lshw -class network:\n" >> ./FortinetSupport.log ; sudo lshw -class network >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo lshw -class network -short:\n" >> ./FortinetSupport.log ; sudo lshw -class network -short >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo lsmod:\n" >> ./FortinetSupport.log ; sudo lsmod >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo dmidecode:\n" >> ./FortinetSupport.log ; sudo dmidecode >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo ifconfig -a:\n" >> ./FortinetSupport.log ; sudo ifconfig -a >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo ip addr show:\n" >> ./FortinetSupport.log ; sudo ip addr show >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo ip link show:\n" >> ./FortinetSupport.log ; sudo ip link show >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo ip -s link show:\n" >> ./FortinetSupport.log ; sudo ip -s link show >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo ip route:\n" >> ./FortinetSupport.log ; sudo ip route >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo route -n:\n" >> ./FortinetSupport.log ; sudo route -n >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo ip -br link show:\n" >> ./FortinetSupport.log ; sudo ip -br link show >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo ip -br addr show:\n" >> ./FortinetSupport.log ; sudo ip -br addr show >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo nmcli connection show:\n" >> ./FortinetSupport.log ; sudo nmcli connection show >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo nmcli device show:\n" >> ./FortinetSupport.log ; sudo nmcli device show >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo VBoxManage list vms:\n" >> ./FortinetSupport.log ; sudo VBoxManage list vms >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo virsh list --all:\n" >> ./FortinetSupport.log ; sudo sudo virsh list --all >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo docker ps -a:\n" >> ./FortinetSupport.log ; sudo docker ps -a >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo vmware-cmd -l:\n" >> ./FortinetSupport.log ; sudo sudo vmware-cmd -l >> ./FortinetSupport.log ; \ echo -e "\n\n###Output of sudo lxc-ls --fancy:\n" >> ./FortinetSupport.log ; sudo lxc-ls --fancy >> ./FortinetSupport.log ; \ sudo ls -lahR /opt/forticlientems/* > ./ls_opt_forticlientems.log ; \ sudo ls -lahR /etc/systemd/* > ./ls_etc_systemd.log ; \ sudo ls -lahR /var/lib/postgresql/* > ./ls_var_lib_postgress.log ; \ sudo journalctl -u ufw --since -15d > ./ufw_logs.txt ; \ sudo journalctl -x --since -15d > ./all_journal_logs.txt ; \ sudo systemctl --all status > ./all_systemctl_logs.txt ; \ sudo dmesg > ./dmesg_logs.txt ; \ tar -czvf ./all_logs.tar.gz ./FortinetSupport.log ./ls_opt_forticlientems.log ./ls_etc_systemd.log ./ufw_logs.txt ./all_journal_logs.txt ./all_systemctl_logs.txt ./ls_var_lib_postgress.log ./dmesg_logs.txt \ /var/log/installer/ /var/log/apt/ /etc/apache2/ /etc/cron* /etc/apt/ /etc/postgresql/ /var/log/pgagent/ /var/log/unattended-upgrades/ /var/log/forticlientems/ /var/log/apache2/ /var/log/postgresql/ /var/log/redis/ /var/log/syslog/ /var/log/auth* /var/log/dpkg* ; \ ls -lath