Troubleshooting Tip: SCP 'auto-backup'

[FMG_TAC_Eng_1]

Description

Configuring SCP auto-backup for FortiManager and FortiAnalyzer and performing basic troubleshooting.

This article would not explore troubleshooting methods for SSH server on receiving end of the session. 

Scope

FortiManager, FortiAnalyzer

Solution

Example of FortiManager settings that will create a backup copy of system configuration.

Destination server – Linux server with enabled SSH access and IP address 10.109.21.14

Source server – FortiManager 7.0.3, IP address 10.109.21.205.

FortiManager CLI config for scheduled configuration backup:

# config system backup all-settings

    set status enable

    set server <ip address of the linux server above>

    set user <username>

    set directory <backup directory on server>

    set week_days monday tuesday …

    set time <time>

    set protocol scp

    set cert <cert-name>

end

Troubleshooting FortiManager/FortiAnalyzer scheduled backup using CLI commands:

It is important to make sure that FortiManager/FortiAnalyzer does not have file system errors:

# diagnose debug klog

If in a command output, it is possible to find messages like 'EXT4-fs error'  first complete file system check with command:

# diagnose system fsck harddisk

Executing this command will restart FortiManager/FortiAnalyzer.

System reboot will also solve a problem with a lack of space in /tmp folder, which can happen if processes were stopped unexpectedly and they had no chance to clear up temporary files.

It is possible to check if /tmp is full with CLI command:

# diagnose system print df

After completing basic system observation, it is possible to proceed to SSH client specific debugs:

# diagnose debug rese

# diagnose debug application ssh

# diagnose debug enabl

After launching a debug it is possible ti change automatic backup schedule in CLI and set it to start in 2 minutes.

!!! When debug session is finished it is important to disable logging in CLI !!!

# diagnose debug disabl

# diagnose debug rese

Example output when certificates were configured incorrectly:

Successful communication start, before processing authentication:

debug2: ssh_connect_direct

debug1: Connecting to 10.109.21.14 [10.109.21.14] port 22.

debug1: Connection established.

debug1: identity file /etc/cert/ssh/fmg_backup type -1

debug1: identity file /etc/cert/ssh/fmg_backup-cert type -1

debug1: Local version string SSH-2.0-OpenSSH_8.1

debug1: Remote protocol version 2.0, remote software version OpenSSH_7.3-hpn14v5 FreeBSD-openssh-portable-7.3.p1_1,1

debug1: match: OpenSSH_7.3-hpn14v5 FreeBSD-openssh-portable-7.3.p1_1,1 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002

debug2: fd 3 setting O_NONBLOCK

debug1: Authenticating to 10.109.21.14:22 as 'root'

debug3: hostkeys_foreach: reading file "/.ssh/known_hosts"

debug3: record_hostkey: found key type ECDSA in file /.ssh/known_hosts:1

debug3: load_hostkeys: loaded 1 keys from 10.109.21.14

debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521

debug3: send packet: type 20

debug1: SSH2_MSG_KEXINIT sent

debug3: receive packet: type 20

debug1: SSH2_MSG_KEXINIT received

debug2: local client KEXINIT proposal

Failing to authenticate on a target server with chosen certificate:

debug3: authmethod_lookup publickey

debug3: remaining preferred: keyboard-interactive,password

debug3: authmethod_is_enabled publickey

debug1: Next authentication method: publickey

debug1: Trying private key: /etc/cert/ssh/fmg_backup

debug3: sign_and_send_pubkey: RSA SHA256:Rl64SmxOmI3+A4G0u+5BA20rwqly8dE7lok8nxJWEvY

debug3: sign_and_send_pubkey: signing using rsa-sha2-512

debug3: send packet: type 50

debug2: we sent a publickey packet, wait for reply

debug3: receive packet: type 51

debug1: Authentications that can continue: publickey,password

debug2: we did not send a packet, disable method

debug3: authmethod_lookup password

debug3: remaining preferred: ,password

debug3: authmethod_is_enabled password

debug1: Next authentication method: password

=================================================

In this context Msg type 51 is interpreted as SSH_MSG_USERAUTH_FAILURE, this value is returned by SSH server.

https://www.iana.org/assignments/ssh-parameters/ssh-parameters.xhtml References standard return codes issued by SSH server.

On a destination server following logs can be referenced:

/var/log/auth.log – to identify authentication errors

/var/log/daemon.log – to identify sshd specific errors

It is recommended to search for SSHD specific entries.

Location of files, filenames, level of logged details can vary greatly between different Linux builds and settings used for daemons.

FortiManager/FortiAnalyzer Event Log messages related to SCP/SFTP/FTP backup process:

Go to System Settings -> Event Log to view the local log list.

FortiManager/FortiAnalyzer logs can be filtered by operation type:

operation="system backup"

Resulting list of event messages will have basic information on successful and failed attempts, with limited details.

Information required for submitting tickets:

- Communication scheme between FortiManager/FortiAnalyzer and backup server.

- Results of executing command '# execute tac report' on FortiManager/FortiAnalyzer.

- Events from section 'FortiManager/FortiAnalyzer Event Log messages related to SCP/SFTP/FTP backup process'.

- Logs from section 'Troubleshooting FortiManager/FortiAnalyzer scheduled backup using CLI commands'.

Related articles:

Technical Tip: How to create a log file of a session using PuTTY

https://community.fortinet.com/t5/FortiGate/Technical-Tip-How-to-create-a-log-file-of-a-session-usin...

Technical Tip: How to stop a long backup job on the FortiAnalyzer

https://community.fortinet.com/t5/FortiAnalyzer/Technical-Tip-How-to-stop-a-long-backup-job-on-the-F...

Technical Tip: Configure SCP backup

https://community.fortinet.com/t5/FortiAnalyzer/Technical-Tip-Configure-SCP-backup/ta-p/194344

Technical Tip: How to send automatic backup configuration

https://community.fortinet.com/t5/FortiAnalyzer/Technical-Tip-How-to-send-automatic-backup-configura...