Netmiko: FileTransfer - ProxyCommandFailure

[usunnapu]

Hi Kirk,

I have written a automation taking asa_upgrade as a reference. While doing a file copy from source (linux OS) to cisco asa, I can see the file is copied but the log file is filled with messages as below

Instead of directly connecting to the switch, I am using a proxy file which contains information about the jump host, ip address etc.

ERROR:paramiko.transport:Exception: ('ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no <user>@<linux host> nc 192.168.100.100 22', 'Broken pipe')
ERROR:paramiko.transport:Traceback (most recent call last):
ERROR:paramiko.transport:  File "<workspace>/runtime/lib/python2.7/site-packages/paramiko/transport.py", line 1766, in run
ERROR:paramiko.transport:    self._channel_handler_table[ptype](chan, m)
ERROR:paramiko.transport:  File "<workspace>/runtime/lib/python2.7/site-packages/paramiko/channel.py", line 1065, in _handle_close
ERROR:paramiko.transport:    self.transport._send_user_message(m)
ERROR:paramiko.transport:  File "<workspace>/runtime/lib/python2.7/site-packages/paramiko/transport.py", line 1586, in _send_user_message
ERROR:paramiko.transport:    self._send_message(data)
ERROR:paramiko.transport:  File "<workspace>/runtime/lib/python2.7/site-packages/paramiko/transport.py", line 1566, in _send_message
ERROR:paramiko.transport:    self.packetizer.send_message(data)
ERROR:paramiko.transport:  File "<workspace>/runtime/lib/python2.7/site-packages/paramiko/packet.py", line 364, in send_message
ERROR:paramiko.transport:    self.write_all(out)
ERROR:paramiko.transport:  File "<workspace>/runtime/lib/python2.7/site-packages/paramiko/packet.py", line 284, in write_all
ERROR:paramiko.transport:    n = self.__socket.send(out)
ERROR:paramiko.transport:  File "<workspace>/runtime/lib/python2.7/site-packages/paramiko/proxy.py", line 71, in send
ERROR:paramiko.transport:    raise ProxyCommandFailure(' '.join(self.cmd), e.strerror)

any idea as to what might be causing this issue above.

[ktbyers]

How are you doing the SSH proxy integration? Can you show the code for this (the ConnectHandler part) and your SSH config file (you can obscure any confidential information).

The Secure Copy probably won't work using Netmiko and SSH proxy.

[usunnapu]

ConnectHandler Part

       def ssh_config_file(self):
        ssh_config = NamedTemporaryFile(delete=False)
        content = ("host {}\n"
                           "    hostname {}\n"
                           "    user <username>\n"
                           "    Port 1022\n"
                           "    StrictHostKeyChecking no\n"
                           "    UserKnownHostsFile /dev/null\n"
                           "    KexAlgorithms +diffie-hellman-group1-sha1\n"
                           "    ProxyCommand ssh -o 'UserKnownHostsFile=/dev/null' "
                           "-o 'StrictHostKeyChecking=no' "
                           "<linux username>r@{} nc %h %p\n".format(self.ip, self.ip, self.jumphost))

        with open('{}'.format(proxy_file.name), 'w+') as proxy_config:
            proxy_config.write(content)

        self.proxyfile = ssh_config.name

    def connect_to_network_device(self):
        user_credential = '<password>'
        try:
            net_device = {
                'device_type': '{}'.format(self.device_type),
                'host': '{}'.format(self.ip),
                'username': '<username>',
                'password': '{}'.format(user_credential),
                'secret': '{}'.format(user_credential),
                'ssh_config_file': '{}'.format(self.proxyfile),
            }

            return ConnectHandler(**net_device)

        except NetMikoTimeoutException:
            logging.error('Netmiko Timeout: {}'.format(self.ip))
        except NetMikoAuthenticationException:
            logging.error('Netmiko Authentication Error: {}'.format(self.ip))
        except SSHException:
            logging.error('Error reading SSH Protocol banner: {}'.format(self.ip))
        except ValueError:
            logging.info('router prompt not valid')

Also the other thing I noticed is, When I was trying to use below pieces of code, I am getting errors as well.

                # if not scp.verify_space_available():
                #     raise ValueError('Insufficient space available')

                # if scp.verify_file():
                #     logging.info('source and destination image MD5 matches')
                # else:
                #     raise ValueError('source and destination image MD5 failed')

Error as below

 File "<workspace>/runtime/lib/python2.7/site-packages/netmiko/scp_handler.py", line 87, in __exit__
    raise exc_type(exc_value)
AttributeError: 'NoneType' object has no attribute 'group'

If I comment out the scp.verify_space_available() and scp.verify_file(), then I dont see the error.

FYI...without using SSH proxy, I was able to test this successfully and none of the above mentioned problems exist. Seems like you said some issue with using Network/SSH Config along with SCP.

[ktbyers]

Can you post your finalized SSH config file (i.e. the file that you write out)? It looks like your SSH config file is not valid (at least from a Paramiko perspective).

If you use this file manually from the Linux command-line for SSH does the SSH proxy work? Note, you will have to use inline_transfer to use this for file transfer and it won't work for Secure Copy (and inline transfer only works for Cisco IOS).

You should probably open up a separate issue on the verify_space_available and verify_file items, but I think we should probably work on this issue first.

[usunnapu]

Hi Kirk,

host 192.168.100.100
      hostname 192.168.100.100
      user <network device username>
      Port <port>
      StrictHostKeyChecking no
      UsersKnownHostsFile /dev/null
      KexAlgorithms +diffie-hellman-group1-sha1
      ProxyCommand ssh -o 'UserKnownHostsFile=/dev/null' -o 'StrictHostKeyChecking=no' <linux user>@<jumphost-ip> nc %h %p 

I am able to connect to the device with the above proxy config file. Issue pops up only when I am using the SCP/FileTransfer functionality.

Also about inline_transfer that you mentioned. If you can provide me an example, I will give that a try as well to see if that fixes the issue.

Let me know if you want me to try with a different proxy file setup.

[ktbyers]

So Netmiko works with this proxy configuration for just SSH?

[ktbyers]

@usunnapu In your file_transfer function call, you should be able to set inline_transfer=True for 'cisco_ios' devices.

def file_transfer(ssh_conn, source_file, dest_file, file_system=None, direction='put',
                  disable_md5=False, inline_transfer=False, overwrite_file=False):

[ktbyers]

This will no longer use SCP instead it will use TCL for the file-transfer (TCL on the IOS-box). It won't work for binary files only for text-files so it won't work for image transfers.

[usunnapu]

So Netmiko works with this proxy configuration for just SSH?

Netmiko Works.

[usunnapu]

@usunnapu In your file_transfer function call, you should be able to set inline_transfer=True for 'cisco_ios' devices.

def file_transfer(ssh_conn, source_file, dest_file, file_system=None, direction='put',
                  disable_md5=False, inline_transfer=False, overwrite_file=False):

I will check this up. But we are having some limitation here and are forced to use Netmiko version 1.4 I believe. So when I tried to do from netmiko import file_transfer I hit an error like ImportError: Cannot import name file_transfer. However, from netmiko import FileTransfer works.

As long as it works for CISCO devices for the time being, I am good.

[usunnapu]

This will no longer use SCP instead it will use TCL for the file-transfer (TCL on the IOS-box). It won't work for binary files only for text-files so it won't work for image transfers.

I missed this point there. So if I want to copy some firewall image, this file_transfer wont work ??

[ktbyers]

Ooops, I missed/forgot your original problem statement i.e. that you were doing OS upgrades. I don't have any solution for that combination of problems i.e. :

1. SSH Proxy is required.
2. You want to do a Secure Copy of the image file.

You would probably have to try to create a custom solution.

[usunnapu]

Thanks Kirk for your time. Just before you close out this, do you see by any chance this in your support plan ?

[ktbyers]

I doubt it, but feel free to come up with a solution for it and submit a PR on it. It is just very low on my set of priorities.

[usunnapu]

Will do. Closing this issue now.

[ktbyers]

If there is an easy way to bind the Secure Copy transfer to the SSH config file and consequently the SSH proxy, I would be totally open to it.

I really haven't looked at it (much).

I will open an issue on it so it is at least tracked as an 'enhancement'

[ktbyers]

https://github.com/ktbyers/netmiko/issues/1033