Netmiko: Nokia SR OS/Alcatel AOS - randomly returned ValueError: router prompt not found

[akei9]

I have a problem with randomly returned ValueError and _router prompt not found_ message for Alcatel/Nokia devices.
Error occurs totally randomly. When I'm running my script once again after error occurs everything goes fine.

Part of my connection settings (I noticed that the only changes I made before the problem started was increasing the following parameters value):

 connection_credentials = {
           [...]
           "banner_timeout": 60,
           "timeout": 300,
           "session_timeout": 180
}

Stack trace:

[...]
  File "/path/to/file/file.py", line 54, in connect_to_host
    self.connection = ConnectHandler(**connection_credentials)
  File "/usr/local/lib/python3.6/site-packages/netmiko/ssh_dispatcher.py", line 259, in ConnectHandler
    return ConnectionClass(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 327, in __init__
    self._open()
  File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 333, in _open
    self._try_session_preparation()
  File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 756, in _try_session_preparation
    self.session_preparation()
  File "/usr/local/lib/python3.6/site-packages/netmiko/nokia/nokia_sros_ssh.py", line 38, in session_preparation
    self.set_base_prompt()
  File "/usr/local/lib/python3.6/site-packages/netmiko/nokia/nokia_sros_ssh.py", line 52, in set_base_prompt
    cur_base_prompt = super().set_base_prompt(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/netmiko/base_connection.py", line 1081, in set_base_prompt
    raise ValueError(f"Router prompt not found: {repr(prompt)}")
ValueError: Router prompt not found: 'A:'

Logs when the problem doesn't occur:

09/17/2020 09:30:38:DEBUG:paramiko.transport:userauth is OK
09/17/2020 09:30:38:INFO:paramiko.transport:Authentication (password) successful!
09/17/2020 09:30:38:DEBUG:paramiko.transport:[chan 0] Max packet in: 32768 bytes
09/17/2020 09:30:38:DEBUG:paramiko.transport:[chan 0] Max packet out: 32768 bytes
09/17/2020 09:30:38:DEBUG:paramiko.transport:Secsh channel 0 opened.
09/17/2020 09:30:38:DEBUG:paramiko.transport:[chan 0] Sesch channel 0 request ok
09/17/2020 09:30:38:DEBUG:paramiko.transport:[chan 0] Sesch channel 0 request ok
09/17/2020 09:30:39:DEBUG:netmiko:read_channel: ^M
^M
A:HOST1#
09/17/2020 09:30:39:DEBUG:netmiko:read_channel:
09/17/2020 09:30:41:DEBUG:netmiko:read_channel:
09/17/2020 09:30:41:DEBUG:netmiko:read_channel:
09/17/2020 09:30:41:DEBUG:netmiko:write_channel: b'\n'
09/17/2020 09:30:41:DEBUG:netmiko:read_channel: ^M
A:HOST1#
09/17/2020 09:30:41:DEBUG:netmiko:read_channel:
09/17/2020 09:30:41:DEBUG:netmiko:[find_prompt()]: prompt is A:HOST1#
09/17/2020 09:30:42:DEBUG:netmiko:read_channel:
09/17/2020 09:30:42:DEBUG:netmiko:In disable_paging
09/17/2020 09:30:42:DEBUG:netmiko:Command: environment no more

Logs when the problem occurs:

09/17/2020 09:42:16:DEBUG:paramiko.transport:userauth is OK
09/17/2020 09:42:17:INFO:paramiko.transport:Authentication (password) successful!
09/17/2020 09:42:17:DEBUG:paramiko.transport:[chan 0] Max packet in: 32768 bytes
09/17/2020 09:42:17:DEBUG:paramiko.transport:[chan 0] Max packet out: 32768 bytes
09/17/2020 09:42:17:DEBUG:paramiko.transport:Secsh channel 0 opened.
09/17/2020 09:42:17:DEBUG:paramiko.transport:[chan 0] Sesch channel 0 request ok
09/17/2020 09:42:17:DEBUG:paramiko.transport:[chan 0] Sesch channel 0 request ok
09/17/2020 09:42:18:DEBUG:netmiko:read_channel:

A:HOST2#
09/17/2020 09:42:18:DEBUG:netmiko:read_channel:
09/17/2020 09:42:20:DEBUG:netmiko:read_channel:
09/17/2020 09:42:20:DEBUG:netmiko:read_channel:
09/17/2020 09:42:20:DEBUG:netmiko:write_channel: b'\n'
09/17/2020 09:42:20:DEBUG:netmiko:read_channel:
A:
09/17/2020 09:42:20:DEBUG:netmiko:read_channel: HOST2#
09/17/2020 09:42:20:DEBUG:netmiko:Clear buffer detects data in the channel
09/17/2020 09:42:21:DEBUG:netmiko:read_channel:
09/17/2020 09:42:21:DEBUG:netmiko:[find_prompt()]: prompt is A:
09/17/2020 09:42:21:DEBUG:netmiko:write_channel: b'logout\n'
09/17/2020 09:42:21:DEBUG:paramiko.transport:EOF in transport thread

[ktbyers]

Are these physical devices or virtual devices?

It looks like the failure occurred when the read for find_prompt() is only getting part of the data back i.e.: A:.

Have you tried setting global_delay_factor: 2?

[akei9]

Are these physical devices or virtual devices?

These are physical devices.

It looks like the failure occurred when the read for find_prompt() is only getting part of the data back i.e.: A:.

Have you tried setting global_delay_factor: 2?

Yes, I noticed it. And no, I haven't tried it yet. At first I'm trying to find the real cause of this problem. What do you think - which timeout may be the cause?

As I said it's randomly returned error. Moreover, it started after changing the values ​​of the above-mentioned timeouts to larger ones.

[akei9]

Still have this problem. I'm using version 3.1.0 and I've see that in 3.2.0 release is added new conn_timeout argument. Maybe it should help?

[akei9]

Still have this problem. I'm using version 3.1.0 and I've see that in 3.2.0 release is added new conn_timeout argument. Maybe it should help?

FYI - I've tried it and it doesn't help.

[cummings-chris]

Hello there, I am also running into this issue with SR-OS and netmiko complaining about the router prompt. I have tried this with and without the global_delay_factor set to 2 and I still get the same issue.

Here is my traceback:

$ python netmiko_test.py
Traceback (most recent call last):
  File "netmiko_test.py", line 18, in <module>
    net_connect = Netmiko(host='localhost', username='admin', password='admin', port='8888', device_type='alcatel_sros')
  File "/Users/chriscummings/Documents/Code/textfsm_playground/venv/lib/python3.8/site-packages/netmiko/ssh_dispatcher.py", line 302, in ConnectHandler
    return ConnectionClass(*args, **kwargs)
  File "/Users/chriscummings/Documents/Code/textfsm_playground/venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 344, in __init__
    self._open()
  File "/Users/chriscummings/Documents/Code/textfsm_playground/venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 350, in _open
    self._try_session_preparation()
  File "/Users/chriscummings/Documents/Code/textfsm_playground/venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 775, in _try_session_preparation
    self.session_preparation()
  File "/Users/chriscummings/Documents/Code/textfsm_playground/venv/lib/python3.8/site-packages/netmiko/nokia/nokia_sros_ssh.py", line 40, in session_preparation
    self.set_base_prompt()
  File "/Users/chriscummings/Documents/Code/textfsm_playground/venv/lib/python3.8/site-packages/netmiko/nokia/nokia_sros_ssh.py", line 62, in set_base_prompt
    cur_base_prompt = super().set_base_prompt(*args, **kwargs)
  File "/Users/chriscummings/Documents/Code/textfsm_playground/venv/lib/python3.8/site-packages/netmiko/base_connection.py", line 1141, in set_base_prompt
    raise ValueError(f"Router prompt not found: {repr(prompt)}")
ValueError: Router prompt not found: 'A:lab-cr1'

And here is my debug log:

DEBUG:paramiko.transport:starting thread (client mode): 0x6feef10
DEBUG:paramiko.transport:Local version/idstring: SSH-2.0-paramiko_2.7.2
DEBUG:paramiko.transport:Remote version/idstring: SSH-2.0-OpenSSH_3.5p1
INFO:paramiko.transport:Connected (version 2.0, client OpenSSH_3.5p1)
DEBUG:paramiko.transport:kex algos:['diffie-hellman-group-exchange-sha1', 'diffie-hellman-group1-sha1'] server key:['ssh-rsa', 'ssh-dss'] client encrypt:['aes256-ctr', 'aes192-ctr', 'aes128-ctr', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'arcfour', 'aes192-cbc', 'aes256-cbc', '[email protected]'] server encrypt:['aes256-ctr', 'aes192-ctr', 'aes128-ctr', 'aes128-cbc', '3des-cbc', 'blowfish-cbc', 'cast128-cbc', 'arcfour', 'aes192-cbc', 'aes256-cbc', '[email protected]'] client mac:['hmac-md5', 'hmac-sha1', 'hmac-ripemd160', '[email protected]', 'hmac-sha1-96', 'hmac-md5-96'] server mac:['hmac-md5', 'hmac-sha1', 'hmac-ripemd160', '[email protected]', 'hmac-sha1-96', 'hmac-md5-96'] client compress:['none'] server compress:['none'] client lang:[''] server lang:[''] kex follows?False
DEBUG:paramiko.transport:Kex agreed: diffie-hellman-group-exchange-sha1
DEBUG:paramiko.transport:HostKey agreed: ssh-rsa
DEBUG:paramiko.transport:Cipher agreed: aes128-ctr
DEBUG:paramiko.transport:MAC agreed: hmac-sha1
DEBUG:paramiko.transport:Compression agreed: none
DEBUG:paramiko.transport:Got server p (1024 bits)
DEBUG:paramiko.transport:kex engine KexGex specified hash_algo <built-in function openssl_sha1>
DEBUG:paramiko.transport:Switch to new keys ...
DEBUG:paramiko.transport:Adding ssh-rsa host key for [localhost]:8888: b'a3e473421d66489242c7312d42e469c5'
DEBUG:paramiko.transport:userauth is OK
INFO:paramiko.transport:Authentication (password) successful!
DEBUG:paramiko.transport:[chan 0] Max packet in: 32768 bytes
DEBUG:paramiko.transport:[chan 0] Max packet out: 32768 bytes
DEBUG:paramiko.transport:Secsh channel 0 opened.
DEBUG:paramiko.transport:[chan 0] Sesch channel 0 request ok
DEBUG:paramiko.transport:[chan 0] Sesch channel 0 request ok
DEBUG:netmiko:read_channel: 
This simulation software is the property of Alcatel. It may not
be sold, leased, loaned or licensed to any non-Alcatel entity.
A:lab-cr1# 
DEBUG:netmiko:read_channel: 
DEBUG:netmiko:read_channel: 
DEBUG:netmiko:read_channel: 
DEBUG:netmiko:write_channel: b'\n'
DEBUG:netmiko:read_channel: 
DEBUG:netmiko:read_channel: 
DEBUG:netmiko:write_channel: b'\n'
DEBUG:netmiko:read_channel: 
A:lab-cr1
DEBUG:netmiko:read_channel: # 
A:lab-cr1# 
DEBUG:netmiko:Clear buffer detects data in the channel
DEBUG:netmiko:read_channel: 
DEBUG:netmiko:[find_prompt()]: prompt is A:lab-cr1
DEBUG:netmiko:write_channel: b'logout\n'

Any help would be greatly appreciated, and I can test more if needed. (I am pretty new to netmiko, however, so please bear with me 😄 ) Thank you!

[ktbyers]

Okay, it seems like the generally issue is that in some situations find_prompt() is failing to get the last character i.e. the # and that is causing problems.

I will have to look into why that is happening.

[ktbyers]

Adding @hellt also.

I will probably look at this, but it might be next week. Not sure I am going to get to it this week.