Netmiko: Timeouts are not directly configurable

[xavierhardy]

In most of the code, timeouts are handled as number of iterations (which are not always configurable) and depend on the delay factor. Wouldn't it be easier to be able to set the timeouts in terms of seconds (floating)? Here is an example:

# Check if the only thing you received was a newline
count = 0
prompt = prompt.strip()
while count <= 10 and not prompt:
    prompt = self.read_channel().strip()
    if prompt:
        if self.ansi_escape_codes:
            prompt = self.strip_ansi_escape_codes(prompt).strip()
    else:
        self.write_channel(self.RETURN)
        time.sleep(delay_factor * .1)
    count += 1

Diminishing the delay_factor might make the code fail as the timeout is also reduced. I see the delay factor more as a polling interval, it should be configurable without affecting the timeout. It basically means that decreasing it will make the process use more CPU time and nothing else. One could even give a whole CPU to netmiko by setting it 0.

Here is an example of a working replacement:

# The timeout should be given as an argument (not the case in my example obviously)
# it should be configurable globally from the connection handler
# 11 seconds is the original timeout as the delay_factor defaults to 1, and there are 11 iterations
timeout = 11
# Check if the only thing you received was a newline
prompt = prompt.strip()
start = time.time()
while time.time() - start < timeout and not prompt:
    prompt = self.read_channel().strip()
    if prompt:
        if self.ansi_escape_codes:
            prompt = self.strip_ansi_escape_codes(prompt).strip()
    else:
        self.write_channel(self.RETURN)
        time.sleep(delay_factor * .1)

[xavierhardy]

The best solution would be to use 'select' to poll for messages.

[ktbyers]

@xavierhardy Yeah, maybe (on select).

Netmiko is really optimized for reliability not performance. I am pretty certain that is the right choice to make (i.e. in conflicts between reliability and performance...choose reliability over being faster). Being slow is

...that being said, there could be hard-coded delays that maybe should be affected by global_delay_factor.

Note, the primary purpose of global_delay_factor/delay_factor is to go slower, not faster.

Also it is possible that using select is fundamentally a better solution (that was proposed by someone else a while back).

I am open to hearing a more on using select.

Note, it is a lot of work to re-write this code and to re-test/re-validate it (and it will come at the expense of writing other things so there is also a question of whether it is marginally worth it or not).

[xavierhardy]

I saw you have written an issue about unit tests (functional tests too?). I can always create PoC, but I wouldn't dare merge it without them.

According to me, the fact that the concepts of delay factors/polling intervals and timeouts are linked is misleading and error prone because if you try decrease the delay factors, you may break things (since the timeout decreases).

At the very least, we should make max_loops and delay_factor configurable everywhere (so that anyone can change the timeout).

Doing a small test (not using select, but adding a timeout with 100ms polling time), I managed to get from 4s to connect and execute 'ls' to 600ms, almost 8x speed up.

[ktbyers]

Yes, understood, but Netmiko supports 44 diverse platforms and I only have access to about 7 to 10 of them. So real testing is hard (especially for fundamental-core changes). Mocking also doesn't generally work since most of the issues are related to device interactions.

I will read through what you have proposed more thoroughly. I have only skimmed through it.

delay_factor and global_delay_factor really are generally intended to go slower not faster.

For major tasks, like send_command, the method will complete once the trailing prompt is detected (or the expect_string if specified). The loop delay there is .2 seconds so it is a pretty marginal difference per loop (and the network devices are generally pretty slow on the CLI).

To me 4 seconds is fine...i.e. I care much more for every 100 users how many of them will have it fail or work (than is it 4 seconds or 600mS)

If you are talking outside of core methods (like send_command and send_config_set), then going faster, reliably is probably going to be difficult (note, I have tried to go faster in the past).

...that all being said, I am open to thinking about these core concepts and how they are implemented. I started to try to come up with a better integration in Netmiko 2 (but this is more for ease of use). Basically, I started to move towards timeout controlling the how many loops and the delay per loop in Netmiko 2 (where prior to this timeout in Netmiko really did very little).

This was mostly so I could make it easier for end-users on how to adjust the settings when they ran into devices that failed (i.e. they need to increase how long Netmiko sleeps).

I am also probably open to allowing timeout/delay_factor/global_delay_factor (including with new concepts) to have things go faster...though the defaults will need conservative. Basically if someone wants to go faster and decrease their reliability, they can choose to do so.

[xavierhardy]

I've done a proof of concept using select. It works in my demo use case (connecting to local SSH server on Linux and sending ls).

Select version: 0.868s
Original version (latest commit): 4.471s
5x speed-up

Here is my commit: https://github.com/xavierhardy/netmiko/commit/72722d93101f8c75654e27419701e8ef6a6cd507#diff-356f254b084894f341050b6b9d88e302

More detailed results:
select + timeout based netmiko

Connection durations:
0.858s
0.755s
0.267s
0.758s
0.803s

Command durations   :
0.223s
0.048s
0.083s
0.069s
0.473s

Average connection  : 0.688s
Average command     : 0.179s
Average total       : 0.868s

Original netmiko: commit fc5769dc39d61aa8095b50b74153af18b4635253

Connection durations:
3.934s
3.833s
3.875s
3.831s
3.869s

Command durations   :
0.603s
0.603s
0.603s
0.602s
0.603s

Average connection  : 3.868s
Average command     : 0.603s
Average total       : 4.471s

[goldyfruit]

That is very nice, any plan to integrate this to netmiko master ?
For me the 4s are an issue with networking-generic-switch.

[ktbyers]

@goldyfruit I need to research it more (and it is low on the priority list and this type of change is a large amount of work).

The general issue is Netmiko is optimized for reliability not performance. In the past trying to go faster reduced reliability (which was a bad tradeoff).

Note, the above test data is not a very good test case (i.e. we need to get test data against real network devices which are generally slower than a locally connected Linux server).

Why does the 4s versus 1s matter? Not trolling, just trying to understand the use cases where this is causing issues.

[goldyfruit]

@ktbyers For example I have to run multiple commands not in the same SSH connection, let say:

• Create ACL _(~7s)_
• Add rules to ACL _(~7 per rule x 4)_
• Apply the ACL to a port _(~7)_

All of these commands have to be performed twice _(one time on each switch)_, that is around ~85s to perform everything and to that add the ~5s of the API call.

This is too long for my APIs and I'm facing timeout, I can increase the timeouts but the user experience will be impacted.

[ktbyers]

Okay, that makes sense. Which platform is it on? (Vendor-model)

How many configuration changes are being pushed (i.e. what is the number of lines that are being changed)?

[goldyfruit]

@ktbyers Arista DCS-7050SX

[ktbyers]

@goldyfruit How many configuration lines are you changing roughly (just trying to understand better what is causing the slowness)?

[goldyfruit]

@ktbyers around 17 commands, it could be 21 sometimes depending the amount of rules.

[ktbyers]

Do you have a bunch of show operations intermingled with the config operations? It is strange to me that it would take 85 seconds for that to occur.

It looks like I have a 1/2 second delay per config command so 10 commands should take about 10 seconds (plus some additional overhead for the connection, going into config mode, et cetera).

[goldyfruit]

No show operations, it's 85 seconds for 2 switches.
I have split the commands in 3 groups _(1 per SSH command)_.

ip access-list a-in
statistics per-entry
ip access-list a-out
statistics per-entry
permit tcp any any established
permit udp any range 53 53 any
permit udp any range 67 68 any

ip access-list a-in
permit tcp any any range 22 22



md5-7537762fd78605bd4c48711b3be0076d



interface port-channel 116
no ip access-group in
no ip access-group out
ip access-group a-in in
ip access-group a-out out

[ktbyers]

Which commands are being applied on which switch? And for each switch is it all accomplished in one SSH connection?

Are you using threading or multiprocessing so you can do the two switches concurrently? That will cut the time roughly in half (assuming you aren't).

[goldyfruit]

All these commands are applied on both switches via 3 different SSH connections.
networking-generic-switch works sequentially :/

[ktbyers]

Why 3 SSH connections?

Is network-generic-switch from openstack?

[goldyfruit]

Because the connections happened at 3 different steps.
Yes from OpenStack project :)

[ktbyers]

Okay...that makes sense as far as why it so slow.

It probably would only take about 20 seconds in total (if concurrency was used and only one SSH connection per device).

You can see if can get it to go faster by setting global_delay_factor which is an argument to ConnectHandler to .5 or to .2.

You would also probably need to modify the delay_factor argument into send_config_set() and set it to the same value as you set ConnectHandler to.

I can point you where these modifications need made in the Netmiko source code if you want to experiment with them.

To me this is really fundamentally an openstack issue (most notably you must have concurrency when dealing with SSH connections).

That being said you might be able to cut the Netmiko total time significantly by being more aggressive in reducing sleep times that are embedded into Netmiko.

[goldyfruit]

@ktbyers Thanks, I'll have a look to change the values you mentioned.
I keep you posted.

[goldyfruit]

@ktbyers way better, ~4s per action.

It could be nice to have the possibility to choose via an option between the select or sleep method, like for example:

• For stability choose the sleep method.
• For performance choose the select method.

I know that is a lot of work to do but just saying :)

[carlmontanari]

Good conversation, but super old so going to close this!