Netmiko: Connection to device timed-out: cisco_ios 10.255.141.240:22

[zweiund40]

Hi Mr. Byers,

I wrote a script using netmiko which connects across a ssh-tunnel to cisco devices gathering output from various commands. The script worked without problems on Ubuntu 16.04. LTS. But after a new installation of 17.10., I'm constantly facing this issue:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/netmiko/base_connection.py", line 392, in establish_connection
    self.remote_conn_pre.connect(**ssh_connect_params)
  File "/usr/lib/python3/dist-packages/paramiko/client.py", line 296, in connect
    to_try = list(self._families_and_addresses(hostname, port))
  File "/usr/lib/python3/dist-packages/paramiko/client.py", line 200, in _families_and_addresses
    addrinfos = socket.getaddrinfo(hostname, port, socket.AF_UNSPEC, socket.SOCK_STREAM)
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/forty2/bin/GetData.py", line 739, in <module>
    GetDataNetmiko(device)
  File "/home/forty2/bin/GetData.py", line 671, in GetDataNetmiko
    net_connect = ConnectHandler(**devstruct)
  File "/usr/lib/python3/dist-packages/netmiko/ssh_dispatcher.py", line 96, in ConnectHandler
    return ConnectionClass(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/netmiko/base_connection.py", line 89, in __init__
    self.establish_connection()
  File "/usr/lib/python3/dist-packages/netmiko/base_connection.py", line 396, in establish_connection
    raise NetMikoTimeoutException(msg)
netmiko.ssh_exception.NetMikoTimeoutException: Connection to device timed-out: cisco_ios 10.255.141.240:22

I tried different things, but nothing worked. It's always the same.

I rely on my .ssh/config. The relevant entries are these:

Host trillian
  User root
  HostName ptlda.net
  Port 42000

Host copper fiber u2-copper u2-fiber 10.255.140.240
  ForwardAgent no
  ForwardX11 no
  KexAlgorithms +diffie-hellman-group1-sha1
  ProxyCommand ssh -q trillian -W %h:%p

Tcpdump shows that there are absolutly no attempts to establish a connection to the jumphost. So "device timed-out" is a little bit misleading.

What is going wrong? Do you have any idea?

Best wishes

[ktbyers]

It looks like it is failing DNS resolution:

socket.gaierror: [Errno -2] Name or service not known

Can you post your code.

[zweiund40]

Yes. It's part of a long script, but I stripped it down to the essential parts:

forty2@bebektor2:~/bin$ cat testnetmiko.py 
#!/usr/bin/python3
#coding: utf-8
#
# (C) by TO
#

import regex
import os
import inspect

from netmiko import ConnectHandler


def GetDataNetmiko():

    devstruct = {
        'device_type':         'cisco_ios',
        #'ip':                  'fiber',
        'ip':                  '10.255.140.240',
        'username':            'user',
        'password':            'pass',
        'ssh_config_file':     '~/.ssh/config',
        'verbose': True,
    }

    net_connect = ConnectHandler(**devstruct)

    output = net_connect.send_command('show users')
    print(output)
    print()
    print(net_connect.find_prompt())
    print()

    net_connect.disconnect()


    return 0



####################
#
#   MAIN
#

if __name__== "__main__":


    GetDataNetmiko()

And from '~.ssh/config'

Host trillian
  User root
  HostName ptlda.net
  Port 42000
  LocalForward 10000 ilosw3.2:23
  LocalForward 10001 ilosw3.4:23
  LocalForward 10002 ilosw10.2:23


Host 10.255.141.240
  ForwardAgent no
  ForwardX11 no
  KexAlgorithms +diffie-hellman-group1-sha1
  ProxyCommand ssh -q trillian -W %h:%p
Host u2-fiber
  ForwardAgent no
  ForwardX11 no
  KexAlgorithms +diffie-hellman-group1-sha1
  ProxyCommand ssh -q trillian -W %h:%p

It works on the command line:

forty2@bebektor2:~/bin$ ssh [email protected]
Password: 

u2-fiber#exit
Connection to 10.255.141.240 closed.


forty2@bebektor2:~/bin$ ssh thoswald@u2-fiber
Password: 

u2-fiber#exit
Connection to u2-fiber closed.
forty2@bebektor2:~/bin$

But not with the script.

[ktbyers]

Can you post the exception stack trace that you get with the code that you just posted above?

[zweiund40]

Yes, I can. But it's the same. And I will do it tomorrow morning. I'm
currently on my way home. It's already 19.30 in Germany. Thx.

Kirk Byers [e-mail] schrieb am Do., 22. März 2018, 18:34:

Can you post the exception stack trace that you get with the code that you
just posted above?

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/ktbyers/netmiko/issues/754#issuecomment-375393354,
or mute the thread
https://github.com/notifications/unsubscribe-auth/Aj6vge03JbSMKCbT_qEV6oywiwoAev7fks5tg-CTgaJpZM4S3aIx
.

[zweiund40]

I simplyfied the script a little bit more.

#!/usr/bin/python3
#coding: utf-8

from netmiko import ConnectHandler


def GetDataNetmiko():

    devstruct = {
        'device_type':         'cisco_ios',
        'ip':                  'u2-fiber',
        #'ip':                  '10.255.141.240',
        'username':            'user',
        'password':            'pass',
        'ssh_config_file':     '/home/forty2/.ssh/config',
        'verbose': True,
    }

    net_connect = ConnectHandler(**devstruct)

    output = net_connect.send_command('show users')
    print(output)
    print()
    print(net_connect.find_prompt())
    print()

    net_connect.disconnect()


    return 0



####################
#
#   MAIN
#

if __name__== "__main__":


    GetDataNetmiko()

Neither of the two ip statements are working. The corresponding /home/forty2/.ssh/config lines:

Host trillian
  User root
  HostName ptlda.net
  Port 42000

Host 10.255.141.240 u2-fiber
  User thoswald
  KexAlgorithms +diffie-hellman-group1-sha1
  ProxyCommand ssh -q trillian -W %h:%p

First test with 'ip':'u2-fiber' script and cli:

forty2@bebektor2:~$ testnetmiko.py 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/netmiko/base_connection.py", line 392, in establish_connection
    self.remote_conn_pre.connect(**ssh_connect_params)
  File "/usr/lib/python3/dist-packages/paramiko/client.py", line 296, in connect
    to_try = list(self._families_and_addresses(hostname, port))
  File "/usr/lib/python3/dist-packages/paramiko/client.py", line 200, in _families_and_addresses
    addrinfos = socket.getaddrinfo(hostname, port, socket.AF_UNSPEC, socket.SOCK_STREAM)
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/forty2/bin/testnetmiko.py", line 42, in <module>
    GetDataNetmiko()
  File "/home/forty2/bin/testnetmiko.py", line 19, in GetDataNetmiko
    net_connect = ConnectHandler(**devstruct)
  File "/usr/lib/python3/dist-packages/netmiko/ssh_dispatcher.py", line 96, in ConnectHandler
    return ConnectionClass(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/netmiko/base_connection.py", line 89, in __init__
    self.establish_connection()
  File "/usr/lib/python3/dist-packages/netmiko/base_connection.py", line 396, in establish_connection
    raise NetMikoTimeoutException(msg)
netmiko.ssh_exception.NetMikoTimeoutException: Connection to device timed-out: cisco_ios u2-fiber:22
forty2@bebektor2:~$ 


forty2@bebektor2:~$ ssh u2-fiber
Password: 

u2-fiber#exit
Connection to u2-fiber closed.
forty2@bebektor2:~$

Second test with 'ip':'10.255.141.240'

forty2@bebektor2:~$ testnetmiko.py 
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/netmiko/base_connection.py", line 392, in establish_connection
    self.remote_conn_pre.connect(**ssh_connect_params)
  File "/usr/lib/python3/dist-packages/paramiko/client.py", line 296, in connect
    to_try = list(self._families_and_addresses(hostname, port))
  File "/usr/lib/python3/dist-packages/paramiko/client.py", line 200, in _families_and_addresses
    addrinfos = socket.getaddrinfo(hostname, port, socket.AF_UNSPEC, socket.SOCK_STREAM)
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/forty2/bin/testnetmiko.py", line 42, in <module>
    GetDataNetmiko()
  File "/home/forty2/bin/testnetmiko.py", line 19, in GetDataNetmiko
    net_connect = ConnectHandler(**devstruct)
  File "/usr/lib/python3/dist-packages/netmiko/ssh_dispatcher.py", line 96, in ConnectHandler
    return ConnectionClass(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/netmiko/base_connection.py", line 89, in __init__
    self.establish_connection()
  File "/usr/lib/python3/dist-packages/netmiko/base_connection.py", line 396, in establish_connection
    raise NetMikoTimeoutException(msg)
netmiko.ssh_exception.NetMikoTimeoutException: Connection to device timed-out: cisco_ios 10.255.141.240:22
forty2@bebektor2:~$ 


forty2@bebektor2:~$ ssh 10.255.141.240
Password: 

u2-fiber#exit
Connection to 10.255.141.240 closed.
forty2@bebektor2:~$

I dumped on the appropriate interface in parallel. No packets with the script tests.

But indeed, there is one obvious behavioral difference. The test with 10.255.141.240 is failing much faster (immediatly) than with u2-fiber.

[ktbyers]

It looks like Paramiko tries to do a getaddrinfo call on both the entries when using SSH config (both in the IP address case and in the u2-fiber case).

Is u2-fiber in DNS or is it relying on the hosts file. Also what is the fully qualified domain name and have you tried using that?

[zweiund40]

Neither one thing nor the other. It‘s fully relying on the .ssh/config. On the originating host and on the jump host. I mean, that‘s how it works on the cli since time immemorial. First .ssh/config, second /etc/hosts, than dns. And only on level of dns, one need the FQDN.

If that‘s the case that paramiko is trying to do that, than there was an unreasonable change in the behavior of paramiko. The originating host has no direct access to the target device u2-fiber. It‘s completely hidden behind trillian. So there is no need for a FQDN on the originating host. I mean, that‘s the way it was working with 16.04.

[ktbyers]

@zweiund40 Yes, good point on the address/name resolution that it shouldn't care.

What happens if you do this manually:

$ python
Python 3.6.2 (default, Feb 19 2018, 21:58:17) 
[GCC 4.8.5 20150623 (Red Hat 4.8.5-11)] on linux
Type "help", "copyright", "credits" or "license" for more information.

>>> import socket
>>> socket.getaddrinfo('10.10.10.10', 22, socket.AF_UNSPEC, socket.SOCK_STREAM)

Replace the 10.10.10.10 with your IP address and also with u2-fiber. That is the class that it looks like Paramiko fails on from the Paramiko source code.

[zweiund40]

forty2@bebektor2:~$ python3
Python 3.6.3 (default, Oct  3 2017, 21:45:48) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> 
>>> import socket
>>> socket.getaddrinfo('10.255.141.240', 22, socket.AF_UNSPEC, socket.SOCK_STREAM)
[(<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 6, '', ('10.255.141.240', 22))]

Immediate reponse.

>>> socket.getaddrinfo('u2-fiber', 22, socket.AF_UNSPEC, socket.SOCK_STREAM)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known
>>>

It lasts roughly 10 seconds until the line comes back.

[ktbyers]

What happens if you change your SSH config file to this and test using an IP address (10.255.141.240)

Host trillian
  User root
  HostName ptlda.net
  Port 42000

Host 10.255.141.240
  User thoswald
  hostname 10.255.141.240
  Port 22
  KexAlgorithms +diffie-hellman-group1-sha1
  ProxyCommand ssh -q trillian -W %h:%p

[ktbyers]

Ooops, must have accidentally clicked the close issue button...

[zweiund40]

Unfortunately the same behavior.

[ktbyers]

Okay, you can restore you SSH config file to what you had previously.

It would be interesting to add a print statement here (to see what it is trying to resolve).

File "/usr/lib/python3/dist-packages/paramiko/client.py", line 200, in _families_and_addresses
    addrinfos = socket.getaddrinfo(hostname, port, socket.AF_UNSPEC, socket.SOCK_STREAM)

Line 200 of that client.py Paramiko file. Make a backup of the file so you can restore it properly after testing.

Just before that do a:

print(hostname)
print(port)

I would be interested in doing this in both cases both the u2-fiber and the IP address case

[zweiund40]

Funny. I just realized the issue was closed. In Germany, we say DFK: "Dicke-Fingerproblem." Literally "thick finger problem." Means accidentially hitting the wrong keys causing weird things.

[zweiund40]

It's meant funny.

[zweiund40]

Uh ... I'm a little bit puzzled now.

ossi@pandora:~/GitSrc$ sudo locate client.py|egrep para
/usr/lib/python2.7/dist-packages/paramiko/client.py

python2.7 ? I expected at least something like python3. I'm not that firm with python. Can that cause the problem?

[ktbyers]

It should be this file here:

/usr/lib/python3/dist-packages/paramiko/client.py

[ktbyers]

This is from your stack trace.

[zweiund40]

Oh! Sorry. I accidentially looked at my Ubuntu 16.04. VM. Sorry.

[zweiund40]

Yes. And there it is:

forty2@bebektor2:~$ sudo locate client.py|egrep para
/usr/lib/python3/dist-packages/paramiko/client.py

[zweiund40]

forty2@bebektor2:~$ testnetmiko.py 
10.255.141.240:22
22
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/netmiko/base_connection.py", line 392, in establish_connection
    self.remote_conn_pre.connect(**ssh_connect_params)
  File "/usr/lib/python3/dist-packages/paramiko/client.py", line 300, in connect
    to_try = list(self._families_and_addresses(hostname, port))
  File "/usr/lib/python3/dist-packages/paramiko/client.py", line 204, in _families_and_addresses
    addrinfos = socket.getaddrinfo(hostname, port, socket.AF_UNSPEC, socket.SOCK_STREAM)
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/forty2/bin/testnetmiko.py", line 42, in <module>
    GetDataNetmiko()
  File "/home/forty2/bin/testnetmiko.py", line 19, in GetDataNetmiko
    net_connect = ConnectHandler(**devstruct)
  File "/usr/lib/python3/dist-packages/netmiko/ssh_dispatcher.py", line 96, in ConnectHandler
    return ConnectionClass(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/netmiko/base_connection.py", line 89, in __init__
    self.establish_connection()
  File "/usr/lib/python3/dist-packages/netmiko/base_connection.py", line 396, in establish_connection
    raise NetMikoTimeoutException(msg)
netmiko.ssh_exception.NetMikoTimeoutException: Connection to device timed-out: cisco_ios 10.255.141.240:22




forty2@bebektor2:~$ testnetmiko.py 
u2-fiber:22
22
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/netmiko/base_connection.py", line 392, in establish_connection
    self.remote_conn_pre.connect(**ssh_connect_params)
  File "/usr/lib/python3/dist-packages/paramiko/client.py", line 300, in connect
    to_try = list(self._families_and_addresses(hostname, port))
  File "/usr/lib/python3/dist-packages/paramiko/client.py", line 204, in _families_and_addresses
    addrinfos = socket.getaddrinfo(hostname, port, socket.AF_UNSPEC, socket.SOCK_STREAM)
  File "/usr/lib/python3.6/socket.py", line 745, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -2] Name or service not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/forty2/bin/testnetmiko.py", line 42, in <module>
    GetDataNetmiko()
  File "/home/forty2/bin/testnetmiko.py", line 19, in GetDataNetmiko
    net_connect = ConnectHandler(**devstruct)
  File "/usr/lib/python3/dist-packages/netmiko/ssh_dispatcher.py", line 96, in ConnectHandler
    return ConnectionClass(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/netmiko/base_connection.py", line 89, in __init__
    self.establish_connection()
  File "/usr/lib/python3/dist-packages/netmiko/base_connection.py", line 396, in establish_connection
    raise NetMikoTimeoutException(msg)
netmiko.ssh_exception.NetMikoTimeoutException: Connection to device timed-out: cisco_ios u2-fiber:22
forty2@bebektor2:~$

[ktbyers]

What version of Netmiko are you using? You can get this from pip list.

[zweiund40]

I always try to avoid pip if it is possible. I rather rely on the official repositories. Maybe I'm wrong, but I think keeping my installation up to date is easier accomplished with apt.

forty2@bebektor2:~$ dpkg -l *netmi*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                               Version                Architecture           Description
+++-==================================-======================-======================-=========================================================================
ii  python-netmiko                     1.1.0-1                all                    multi-vendor library for SSH connections to network devices - Python 2.X
ii  python3-netmiko                    1.1.0-1                all                    multi-vendor library for SSH connections to network devices - Python 3.X

[ktbyers]

That is very old. You need to upgrade to a much newer version.

You should probably get something that is Netmiko >= 2.0

I am pretty sure that is the cause of your problem as that ip_address:port changed in roughly Netmiko 1.4.0.

[zweiund40]

Sigh. Ok, I changed it. I remove the Ubuntu packages and installed it with pip3. And now it works.

forty2@bebektor2:~$ testnetmiko.py 
SSH connection established to u2-fiber:22
Interactive SSH session established
    Line       User       Host(s)              Idle       Location
*  1 vty 0     thoswald   idle                 00:00:00 10.255.140.22

  Interface      User        Mode                     Idle     Peer Address


u2-fiber#

forty2@bebektor2:~$ 



forty2@bebektor2:~$ testnetmiko.py 
SSH connection established to 10.255.141.240:22
Interactive SSH session established
    Line       User       Host(s)              Idle       Location
*  1 vty 0     thoswald   idle                 00:00:00 10.255.140.22

  Interface      User        Mode                     Idle     Peer Address


u2-fiber#

forty2@bebektor2:~$

Many thanks to you, especially that you opened my eyes. Now I think, in case of an error it's always worth to take a closer look to the versions. Many, many thanks.

[zweiund40]

And now I remember. It's several months ago I installed netmiko on my old Ubuntu 16.04. I had to use pip to install it, because netmiko wasn't available in the official Ubuntu repositories. That's why it worked before I installed 17.10. Canonical include it, but unfortunately a very old version. I simply forgot.

[ktbyers]

Yes, I should have asked earlier on, but I didn't really see anything that was correlating to and old Netmiko version until we added the additional print debugging.

[zweiund40]

Oh, sorry! I wasn't aware that I'm the one who is supposed to close the issue.