Netmiko: SCP.py times out too quickly on Secure Copy of large files

[eakikel]

I am using the scp scp_transfer.transfer_file() function.
When running against older versions of EOS like 4.15 thru 4.19, the file transfer works smoothly

When it runs against EOS-4.20.11M the file transfers but errors out
Exception in thread Thread-3:
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv
out = self.in_buffer.read(nbytes, self.timeout)
File "/usr/local/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read
raise PipeTimeout()
paramiko.buffered_pipe.PipeTimeout

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/scp.py", line 335, in _recv_confirm
msg = self.channel.recv(512)
File "/usr/local/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv
raise socket.timeout()
socket.timeout

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/threading.py", line 916, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.6/threading.py", line 864, in run
self._target(self._args, *self._kwargs)
File "./copyarista.py", line 137, in copycode
scp_transfer.transfer_file()
File "/usr/local/lib/python3.6/site-packages/netmiko/scp_handler.py", line 304, in transfer_file
self.put_file()
File "/usr/local/lib/python3.6/site-packages/netmiko/scp_handler.py", line 317, in put_file
self.scp_conn.scp_transfer_file(self.source_file, destination)
File "/usr/local/lib/python3.6/site-packages/netmiko/scp_handler.py", line 40, in scp_transfer_file
self.scp_client.put(source_file, dest_file)
File "/usr/local/lib/python3.6/site-packages/scp.py", line 154, in put
self._send_files(files)
File "/usr/local/lib/python3.6/site-packages/scp.py", line 250, in _send_files
self._send_file(fl, name, mode, size)
File "/usr/local/lib/python3.6/site-packages/scp.py", line 276, in _send_file
self._recv_confirm()
File "/usr/local/lib/python3.6/site-packages/scp.py", line 337, in _recv_confirm
raise SCPException('Timeout waiting for scp response')
scp.SCPException: Timeout waiting for scp response

[ktbyers]

Hmmmmm, strange I am pretty sure I have tested SCP using Netmiko and a similar EOS version as this.

Have you tested it without threading to see if it fails in a simpler use case?

[eakikel]

Removed the threading. Same issue
[kikele@osbaz2dev03087 ~]$ ./testscp.py
File "./testscp.py", line 41
return
^
SyntaxError: 'return' outside function
[kikele@osbaz2dev03087 ~]$ ./testscp.py
nyc-p17-era08-ds *START TRANSFER
Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv
out = self.in_buffer.read(nbytes, self.timeout)
File "/usr/local/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read
raise PipeTimeout()
paramiko.buffered_pipe.PipeTimeout

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/scp.py", line 335, in _recv_confirm
msg = self.channel.recv(512)
File "/usr/local/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv
raise socket.timeout()
socket.timeout

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./testscp.py", line 142, in
scp_transfer.transfer_file()
File "/usr/local/lib/python3.6/site-packages/netmiko/scp_handler.py", line 304, in transfer_file
self.put_file()
File "/usr/local/lib/python3.6/site-packages/netmiko/scp_handler.py", line 317, in put_file
self.scp_conn.scp_transfer_file(self.source_file, destination)
File "/usr/local/lib/python3.6/site-packages/netmiko/scp_handler.py", line 40, in scp_transfer_file
self.scp_client.put(source_file, dest_file)
File "/usr/local/lib/python3.6/site-packages/scp.py", line 154, in put
self._send_files(files)
File "/usr/local/lib/python3.6/site-packages/scp.py", line 250, in _send_files
self._send_file(fl, name, mode, size)
File "/usr/local/lib/python3.6/site-packages/scp.py", line 276, in _send_file
self._recv_confirm()
File "/usr/local/lib/python3.6/site-packages/scp.py", line 337, in _recv_confirm
raise SCPException('Timeout waiting for scp response')
scp.SCPException: Timeout waiting for scp response
[kikele@osbaz2dev03087 ~]$

testscp.txt

[ktbyers]

Okay, strange I checked my lab environment and it is using EOS-4.20.10M and I have no issues.

I read the above error as Paramiko complaining that there was no more data to read from the remote device (and it was expecting more data).

[eakikel]

I have tested a couple of versions of 4.19 with no issues
I have spot checked 3 different version of 4.20 code and they all fail.

Tried setting blocking_timeout to 50 and that was no help

[eakikel]

One other thing. tried smaller file(6000 bytes) and it worked

The files I am having trouble with are 416mb EOS files
Further checks
Works with 115mb file and fails with 131mb file

[eakikel]

TESTED with straight PARAMIKO.
Same problem. So problem is not directly with NETMIKO. It lies with PARAMIKO
Sample code below

[kikele@osbaz2dev03087 ~]$ dd if=/dev/zero of=file.txt count=110 bs=1048576
110+0 records in
110+0 records out
115343360 bytes (115 MB) copied, 0.317923 s, 363 MB/s
[kikele@osbaz2dev03087 ~]$ ssh kikele@nyc-p17-era08-ds del file.txt
Password:
[kikele@osbaz2dev03087 ~]$ ./test-paramiko-scp.py
nyc-p17-era08-ds-nyc-p17-era08-ds COPY WAS Successful
[kikele@osbaz2dev03087 ~]$ dd if=/dev/zero of=file.txt count=125 bs=1048576
125+0 records in
125+0 records out
131072000 bytes (131 MB) copied, 0.403823 s, 325 MB/s
[kikele@osbaz2dev03087 ~]$ ssh kikele@nyc-p17-era08-ds del file.txt
Password:
[kikele@osbaz2dev03087 ~]$ ./test-paramiko-scp.py
nyc-p17-era08-ds-nyc-p17-era08-ds COPY HAS FAILED
[kikele@osbaz2dev03087 ~]$

!/usr/local/bin/python3

import argparse, csv, os, sys, getpass, threading, time, getopt, re, dns.resolver, paramiko, scp
import warnings
warnings.filterwarnings(action='ignore',module='.paramiko.')

name = "nyc-p17-era08-ds"
ip = "nyc-p17-era08-ds"
username = "kikele"
password = "**"
filetocopy = "/home/kikele/file.txt"
from paramiko import SSHClient
from scp import SCPClient
warnings.filterwarnings(action='ignore',module='.paramiko.*')

strip out filename from full path

regex = re.compile(r'.*\/(\S+)')
match = regex.search(filetocopy)
basefile = match.group(1)

print("BASEFILE {0}".format(basefile))

s print("BEFCOPY,{0},{1},{2},{3},{4},{5}".format(name,ip,vendor,model,device_osver,username,password))

try:
ssh = SSHClient()

ssh.load_system_host_keys()

ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

ssh.load_host_keys(os.path.expanduser(os.path.join("~", ".ssh", "known_hosts")))

ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

ssh.connect(ip, username=username, password=password)
with SCPClient(ssh.get_transport()) as scp:
    scp.put(filetocopy, "/mnt/flash/{0}".format(basefile))    

    print("{0}-{1} \tCOPY WAS Successful".format(name,ip))

except:
print("{0}-{1} \tCOPY HAS FAILED".format(name,ip))
ssh.close()

[eakikel]

Further info.
I tracked it down to this issue in the paramiko scp client. Assume you are using that
https://github.com/jbardin/scp.py/issues/103

In my test program, I modified. the following line
adding socket_timeout=40
The time 40 corresponds to file size I am trying to copy
Bigger files need larger time. With 40 I was able to transfer 550mb file
Is there anyway to handle this in netmiko
Thanks

with SCPClient(ssh.get_transport(), socket_timeout=40) as scp:

[ktbyers]

Okay, cool nice find...so it sounds like we just need to wire-in that SCP timeout in some way and increase it to a larger default value than 5 seconds.

[eakikel]

Sounds Good
Thanks

[netengdj]

while this is being fixed, how can we trap this exception? I find that files are actually being transferred fine and something right at the end is generating this. I tried to capture both scp.SCPexception and just SCPexception and neither one is valid.

[netengdj]

so no answer here, is there no way to trap that exception? I see my files are being transferred but even a bare EXCEPT: will not stop this from being raised.

[ktbyers]

@netengdj Are you are using threads or multiple processes?

Catching the exception should be pretty straight-forward i.e. just see the exception generated and wrap it in try/except.

Kirk

[netengdj]

yea I finally realized i have to import that exception specifically
from scp import SCPException

then it was easy to trap. no way i could find to trap it without specifically importing it though.

[eakikel]

👍

Get Outlook for iOShttps://aka.ms/o0ukef

---

From: netengdj [e-mail]
Sent: Tuesday, August 27, 2019 5:10 PM
To: ktbyers/netmiko
Cc: eakikel; Author
Subject: Re: [ktbyers/netmiko] SCP.py times out too quickly on Secure Copy of large files (#1254)

yea I finally realized i have to import that exception specifically
from scp import SCPException

then it was easy to trap. no way i could find to trap it without specifically importing it though.

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHubhttps://github.com/ktbyers/netmiko/issues/1254?email_source=notifications&email_token=ALKJJGA5WD3QKO4PI53VEK3QGWX5ZA5CNFSM4H2VHO4KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD5JMILA#issuecomment-525517868, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ALKJJGGIWVRNESMB5LBF5S3QGWX5ZANCNFSM4H2VHO4A.

[eakikel]

@netengdj Are you are using threads or multiple processes?

Catching the exception should be pretty straight-forward i.e. just see the exception generated and wrap it in try/except.

Kirk

Using threading library
thread = threading.Thread(target=processing_module.sshdevice, args=(parameters,))
thread_list.append(thread)
thread.setDaemon(True)
thread.start()

[carlmontanari]

Closing, see #1397

[eakikel]

copy the 3 modules you changed and tested with new session_timeout set to 60. It works!!

Thanks