Fortigate Backup Issue Kiwi Cattools since 7.0.14

bsl · ‎02-29-2024

After updating various Fortigate models from version 7.0.13 to 7.0.14 the Kiwi Cattools Backup (Device Backup TFTP) brakes. Errorlog: Connection failed (30011) Invalid data received from remote server. Protocol error.

We are using the latest version of Kiwi Cattools (3.12.3.3257).

Maybe someone has the same problem and already found out what the problem is and how to fix it.

Thanks in advance for any help!

bsl · ‎03-08-2024

Good news!

I have the solution to the problem from Solarwinds Support.

I have already tested it and it works perfectly with kiwi cattools 3.12.3.3257.

We have released the Buddy Drop for Cattools to fix the issue with the backup of FortiGate devices.

You may download the Buddy Drop here:  https://downloads.solarwinds.com/solarwinds/Release/PreRelease24/BD/Kiwi-CatTools_3.12.3_BD_KCT-417.zip

It has been tested with Cattools version 3.12.3, the latest, but in theory, it should also work with the previous version.

Below are the details of the BD and the installation/uninstallation steps:

==========================================
SolarWinds Kiwi CatTools 3.12.3 Buddy Drop 
==========================================


This SolarWinds buddy drop addresses the following issue:
   * [Kiwi CatTools] Failed to Backup FortiGate Running FortiOS 7.0.14 


Requirements
============
This buddy drop applies to Kiwi CatTools on the Windows operating system.


Installation instructions
=========================
This buddy drop contains the following files required for installation:

    wodSSH.dll

In the following procedures, the location to install wodSSH.dll is in the following directory:

    C:\Windows\SysWOW64


Install the buddy drop
======================

1. In the KiwiCatTools Manager, stop the Kiwi CatTools service. Shut down all running Kiwi CatTools processes.

2. Back up the following file:

    C:\Windows\SysWOW64\wodSSH.dll

3. Extract the buddy drop archive to a temporary location and copy the wodSSH.dll file. 

4. Replace the wodSSH.dll file with latest wodSSH.dll file in the following directory: 
    
    C:\Windows\SysWOW64\
 
5. Open the Windows command prompt and run as an administrator. Change the directory to C:\Windows\SysWOW64. Register the library file by executing the following command:

       regsvr32 wodSSH.dll 

 The buddy drop is now installed.

6. Open the Kiwi CatTools application and start the Kiwi CatTools service.

7. Run the Activity Device.Running.backup config with the Fortinet device.

  

Result
======

Kiwi CatTools should connect to the Fortinet device and back up the configuration successfully.


Uninstall the buddy drop
========================

1. In the KiwiCatTools Manager, stop the Kiwi CatTools service. Shut down all running Kiwi CatTools processes.

2. Using the wodSSH.dll file you backed up during installation, replace the current wodSSH.dll file.

3. Open the Windows command prompt and run as an administrator. Change the directory to C:\Windows\SysWOW64. Register the library file by executing the following command:

   regsvr32 wodSSH.dll 

 The buddy drop is now uninstalled.

View solution in original post

dbhavsar · ‎02-29-2024

Hello @bsl ,

Looks like the issue is might with the key-offered from your tool. Can you please collect the following debugs:

diagnose debug reset
diagnose debug application sshd -1
diagnose debug cli 8
diagnose debug enable

- Also please try using Putty or alternate SSH Terminal tool.

DNB

tlash35 · ‎03-01-2024

Having a similar issue, see below debug. There are matching keys in the proposal to me.

SSH: fd 7 is not O_NONBLOCK
SSH: Forked child 10639.
SSH: Client protocol version 2.0; client software version WeOnlyDo 3.1.5.244
SSH: no match: WeOnlyDo 3.1.5.244
SSH: Enabling compatibility mode for protocol 2.0
SSH: Local version string SSH-2.0-tWDHoBZYP6GHF
SSH: fd 7 setting O_NONBLOCK
SSH: Proposal: 0, Ciphers: 'diffie-hellman-group-exchange-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521'
SSH: Proposal: 2, Ciphers: 'chacha20-poly1305@openssh.com,aes256-ctr,aes256-gcm@openssh.com'
SSH: Proposal: 3, Ciphers: 'chacha20-poly1305@openssh.com,aes256-ctr,aes256-gcm@openssh.com'
SSH: Proposal: 4, Ciphers: 'hmac-sha2-256,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-512-etm@openssh.com'
SSH: Proposal: 5, Ciphers: 'hmac-sha2-256,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-512-etm@openssh.com'
SSH: list_hostkey_types: rsa-sha2-512,ssh-ed25519
SSH: SSH2_MSG_KEXINIT sent
SSH: SSH2_MSG_KEXINIT received
SSH: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521
SSH: kex_parse_kexinit: rsa-sha2-512,ssh-ed25519
SSH: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes256-ctr,aes256-gcm@openssh.com
SSH: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes256-ctr,aes256-gcm@openssh.com
SSH: kex_parse_kexinit: hmac-sha2-256,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-512-etm@openssh.com
SSH: kex_parse_kexinit: hmac-sha2-256,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-512-etm@openssh.com
SSH: kex_parse_kexinit: none,zlib@openssh.com
SSH: kex_parse_kexinit: none,zlib@openssh.com
SSH: kex_parse_kexinit:
SSH: kex_parse_kexinit:
SSH: kex_parse_kexinit: first_kex_follows 0
SSH: kex_parse_kexinit: reserved 0
SSH: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,curve25519-sha256,curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group14-sha256,diffie-hellman-group18-sha512,diffi
SSH: kex_parse_kexinit: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256
SSH: kex_parse_kexinit: aes128-ctr,aes128-gcm@openssh.com,aes128-cbc,chacha20-poly1305@openssh.com,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-gcm@openssh.com,aes256-ctr,aes256-cbc,rijndael128-cbc,rijndael192-cbc,rijndael256-cbc,rijndael-cbc@lysator.liu
SSH: kex_parse_kexinit: aes128-ctr,aes128-gcm@openssh.com,aes128-cbc,chacha20-poly1305@openssh.com,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-gcm@openssh.com,aes256-ctr,aes256-cbc,rijndael128-cbc,rijndael192-cbc,rijndael256-cbc,rijndael-cbc@lysator.liu
SSH: kex_parse_kexinit: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96,hmac-md5,none
SSH: kex_parse_kexinit: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96,hmac-md5,none
SSH: kex_parse_kexinit: none,none
SSH: kex_parse_kexinit: none,none
SSH: kex_parse_kexinit:
SSH: kex_parse_kexinit:
SSH: kex_parse_kexinit: first_kex_follows 0
SSH: kex_parse_kexinit: reserved 0
SSH: kex: host key algorithm: rsa-sha2-512
SSH: kex: client->server chacha20-poly1305@openssh.com <implicit> none
SSH: kex: server->client chacha20-poly1305@openssh.com <implicit> none
SSH: expecting SSH2_MSG_KEX_ECDH_INIT
SSH: set_newkeys: mode 1
SSH: SSH2_MSG_NEWKEYS sent
SSH: expecting SSH2_MSG_NEWKEYS
SSH: Connection closed by XXX.XXX.CATTOOLSIP.XXX
SSH: This ip XXX.XXX.CATTOOLSIP.XXX is not blocked

Inforeseau · ‎03-04-2024

Hello,
Same issue here with the same versions : Fortigate and Cattools. On the cattools server when using Putty to access console with SSH, it works.

Regards
Jacky

SSH: fd 7 is not O_NONBLOCK
SSH: Forked child 27599.
SSH: Client protocol version 2.0; client software version WeOnlyDo 3.1.5.244
SSH: no match: WeOnlyDo 3.1.5.244
SSH: Enabling compatibility mode for protocol 2.0
SSH: Local version string SSH-2.0-cVJDHEU5
SSH: fd 7 setting O_NONBLOCK
SSH: Proposal: 0, Ciphers: 'diffie-hellman-group-exchange-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521'
SSH: Proposal: 2, Ciphers: 'chacha20-poly1305@openssh.com,aes256-ctr,aes256-gcm@openssh.com'
SSH: Proposal: 3, Ciphers: 'chacha20-poly1305@openssh.com,aes256-ctr,aes256-gcm@openssh.com'
SSH: Proposal: 4, Ciphers: 'hmac-sha2-256,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-512-etm@openssh.com'
SSH: Proposal: 5, Ciphers: 'hmac-sha2-256,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-512-etm@openssh.com'
SSH: list_hostkey_types: rsa-sha2-512,ssh-ed25519
SSH: SSH2_MSG_KEXINIT sent
SSH: SSH2_MSG_KEXINIT received
SSH: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521
SSH: kex_parse_kexinit: rsa-sha2-512,ssh-ed25519
SSH: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes256-ctr,aes256-gcm@openssh.com
SSH: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes256-ctr,aes256-gcm@openssh.com
SSH: kex_parse_kexinit: hmac-sha2-256,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-512-etm@openssh.com
SSH: kex_parse_kexinit: hmac-sha2-256,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-512-etm@openssh.com
SSH: kex_parse_kexinit: none,zlib@openssh.com
SSH: kex_parse_kexinit: none,zlib@openssh.com
SSH: kex_parse_kexinit: 
SSH: kex_parse_kexinit: 
SSH: kex_parse_kexinit: first_kex_follows 0 
SSH: kex_parse_kexinit: reserved 0 
SSH: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,curve25519-sha256,curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group14-sha256,diffie-hellman-group18-sha512,diffi
SSH: kex_parse_kexinit: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256
SSH: kex_parse_kexinit: aes128-ctr,aes128-gcm@openssh.com,aes128-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-gcm@openssh.com,aes256-cbc,rijndael128-cbc,rijndael192-cbc,rijndael256-cbc,rijndael-cbc@lysator.liu.se
SSH: kex_parse_kexinit: aes128-ctr,aes128-gcm@openssh.com,aes128-cbc,aes192-ctr,aes192-cbc,aes256-ctr,aes256-gcm@openssh.com,aes256-cbc,rijndael128-cbc,rijndael192-cbc,rijndael256-cbc,rijndael-cbc@lysator.liu.se
SSH: kex_parse_kexinit: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96,hmac-md5,none
SSH: kex_parse_kexinit: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96,hmac-md5,none
SSH: kex_parse_kexinit: none,none
SSH: kex_parse_kexinit: none,none
SSH: kex_parse_kexinit: 
SSH: kex_parse_kexinit: 
SSH: kex_parse_kexinit: first_kex_follows 0 
SSH: kex_parse_kexinit: reserved 0 
SSH: kex: host key algorithm: rsa-sha2-512
SSH: kex: client->server aes256-ctr hmac-sha2-256-etm@openssh.com none
SSH: kex: server->client aes256-ctr hmac-sha2-256-etm@openssh.com none
SSH: expecting SSH2_MSG_KEX_ECDH_INIT
SSH: set_newkeys: mode 1
SSH: SSH2_MSG_NEWKEYS sent
SSH: expecting SSH2_MSG_NEWKEYS
SSH: Connection closed by XXX.XXX.CATTOOLS_IP.XXX
SSH: This ip XXX.XXX.CATTOOLS_IP.XXX is not blocked

apaulson · ‎03-05-2024

Can confirm we have this issue too! Cattools backups started failing when we upgraded our Fortigates to 7.0.14.

We are on Cattools version 3.12.2.1255

ezhupa · ‎03-06-2024

Hello,

Would you be able to perform the following and test again?
(1.) delete a pre-stored server public key of FGT in SolarWind.
(2.) "execute ssh-regen-keys" on FGT, it regens the host key key file.

From the debugs added so far it seems like the connection is closed by the remote peer but there is no visibility as to the reason for this closure.

bsl · ‎03-07-2024

First of all thanks for your help and sorry for the late feedback.
Here is also the debug

SSH: This ip "KIWICATSERVERIP" is not blocked
SSH: fd 7 is not O_NONBLOCK
SSH: Forked child 17121.
SSH: Client protocol version 2.0; client software version WeOnlyDo 3.1.5.244
SSH: no match: WeOnlyDo 3.1.5.244
SSH: Enabling compatibility mode for protocol 2.0
SSH: Local version string SSH-2.0-IPgP_x0p6qa_aG
SSH: fd 7 setting O_NONBLOCK
SSH: Proposal: 0, Ciphers: 'diffie-hellman-group-exchange-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521'
SSH: Proposal: 2, Ciphers: 'chacha20-poly1305@openssh.com,aes256-ctr,aes256-gcm@openssh.com'
SSH: Proposal: 3, Ciphers: 'chacha20-poly1305@openssh.com,aes256-ctr,aes256-gcm@openssh.com'
SSH: Proposal: 4, Ciphers: 'hmac-sha2-256,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-512-etm@openssh.com'
SSH: Proposal: 5, Ciphers: 'hmac-sha2-256,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-512-etm@openssh.com'
SSH: list_hostkey_types: rsa-sha2-512,ssh-ed25519
SSH: SSH2_MSG_KEXINIT sent
SSH: SSH2_MSG_KEXINIT received
SSH: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521
SSH: kex_parse_kexinit: rsa-sha2-512,ssh-ed25519
SSH: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes256-ctr,aes256-gcm@openssh.com
SSH: kex_parse_kexinit: chacha20-poly1305@openssh.com,aes256-ctr,aes256-gcm@openssh.com
SSH: kex_parse_kexinit: hmac-sha2-256,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-512-etm@openssh.com
SSH: kex_parse_kexinit: hmac-sha2-256,hmac-sha2-256-etm@openssh.com,hmac-sha2-512,hmac-sha2-512-etm@openssh.com
SSH: kex_parse_kexinit: none,zlib@openssh.com
SSH: kex_parse_kexinit: none,zlib@openssh.com
SSH: kex_parse_kexinit: 
SSH: kex_parse_kexinit: 
SSH: kex_parse_kexinit: first_kex_follows 0 
SSH: kex_parse_kexinit: reserved 0 
SSH: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,curve25519-sha256,curve25519-sha256@libssh.org,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group14-sha256,diffie-hellman-group18-sha512,diffi
SSH: kex_parse_kexinit: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss,ssh-ed25519,ecdsa-sha2-nistp521,ecdsa-sha2-nistp384,ecdsa-sha2-nistp256
SSH: kex_parse_kexinit: aes128-ctr,aes128-gcm@openssh.com,aes128-cbc,chacha20-poly1305@openssh.com,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-gcm@openssh.com,aes256-ctr,aes256-cbc,rijndael128-cbc,rijndael192-cbc,rijndael256-cbc,rijndael-cbc@lysator.liu
SSH: kex_parse_kexinit: aes128-ctr,aes128-gcm@openssh.com,aes128-cbc,chacha20-poly1305@openssh.com,3des-cbc,blowfish-cbc,aes192-ctr,aes192-cbc,aes256-gcm@openssh.com,aes256-ctr,aes256-cbc,rijndael128-cbc,rijndael192-cbc,rijndael256-cbc,rijndael-cbc@lysator.liu
SSH: kex_parse_kexinit: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96,hmac-md5,none
SSH: kex_parse_kexinit: hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1,hmac-sha1-96,hmac-md5,none
SSH: kex_parse_kexinit: none,none
SSH: kex_parse_kexinit: none,none
SSH: kex_parse_kexinit: 
SSH: kex_parse_kexinit: 
SSH: kex_parse_kexinit: first_kex_follows 0 
SSH: kex_parse_kexinit: reserved 0 
SSH: kex: host key algorithm: rsa-sha2-512
SSH: kex: client->server chacha20-poly1305@openssh.com <implicit> none
SSH: kex: server->client chacha20-poly1305@openssh.com <implicit> none
SSH: expecting SSH2_MSG_KEX_ECDH_INIT
SSH: set_newkeys: mode 1
SSH: SSH2_MSG_NEWKEYS sent
SSH: expecting SSH2_MSG_NEWKEYS
SSH: Connection closed by "KIWICATSERVERIP"

@ezhupa , thanks for your help, but unfortunately i don't yet know if and how this would be possible with kiwi cattools. i have an open inquiry with solarwinds in this regard.

If i have any news, i will inform you.

bsl · ‎03-08-2024

Good news!

I have the solution to the problem from Solarwinds Support.

I have already tested it and it works perfectly with kiwi cattools 3.12.3.3257.

We have released the Buddy Drop for Cattools to fix the issue with the backup of FortiGate devices.

You may download the Buddy Drop here:  https://downloads.solarwinds.com/solarwinds/Release/PreRelease24/BD/Kiwi-CatTools_3.12.3_BD_KCT-417.zip

It has been tested with Cattools version 3.12.3, the latest, but in theory, it should also work with the previous version.

Below are the details of the BD and the installation/uninstallation steps:

==========================================
SolarWinds Kiwi CatTools 3.12.3 Buddy Drop 
==========================================


This SolarWinds buddy drop addresses the following issue:
   * [Kiwi CatTools] Failed to Backup FortiGate Running FortiOS 7.0.14 


Requirements
============
This buddy drop applies to Kiwi CatTools on the Windows operating system.


Installation instructions
=========================
This buddy drop contains the following files required for installation:

    wodSSH.dll

In the following procedures, the location to install wodSSH.dll is in the following directory:

    C:\Windows\SysWOW64


Install the buddy drop
======================

1. In the KiwiCatTools Manager, stop the Kiwi CatTools service. Shut down all running Kiwi CatTools processes.

2. Back up the following file:

    C:\Windows\SysWOW64\wodSSH.dll

3. Extract the buddy drop archive to a temporary location and copy the wodSSH.dll file. 

4. Replace the wodSSH.dll file with latest wodSSH.dll file in the following directory: 
    
    C:\Windows\SysWOW64\
 
5. Open the Windows command prompt and run as an administrator. Change the directory to C:\Windows\SysWOW64. Register the library file by executing the following command:

       regsvr32 wodSSH.dll 

 The buddy drop is now installed.

6. Open the Kiwi CatTools application and start the Kiwi CatTools service.

7. Run the Activity Device.Running.backup config with the Fortinet device.

  

Result
======

Kiwi CatTools should connect to the Fortinet device and back up the configuration successfully.


Uninstall the buddy drop
========================

1. In the KiwiCatTools Manager, stop the Kiwi CatTools service. Shut down all running Kiwi CatTools processes.

2. Using the wodSSH.dll file you backed up during installation, replace the current wodSSH.dll file.

3. Open the Windows command prompt and run as an administrator. Change the directory to C:\Windows\SysWOW64. Register the library file by executing the following command:

   regsvr32 wodSSH.dll 

 The buddy drop is now uninstalled.

Inforeseau · ‎03-08-2024

Hi bsl,

The solution rocks !!! Many thanks for sharing.

NA_MSI · ‎03-08-2024

Hello, I have been through these steps but still have the same error on 3.12.3.3257, are there any other steps I need to take?