Dvc: py27: dvc push to SSH throws TypeError

[ghost]

Python 2.7
DVC version: 0.52.1

To reproduce:

virtualenv --python=python2.7 .venv
pip install 'dvc[ssh]'
dvc init --no-scm --force
dvc remote add ssh ssh://<your-ssh>
for x in {1..3}; do echo $x > "data/${x}.txt"; done
dvc add data
dvc push -r ssh

Output:

Preparing to upload data to 'ssh://[email protected]/home/mroutis/cache'
Preparing to collect status from ssh://[email protected]/home/mroutis/cache
Collecting information from local cache...
[##############################] 100%

Collecting information from remote cache...
[##############################] 100%
[##############################] 100% Analysing status
[                              ] ?% data/3.txt
ERROR: failed to upload '.dvc/cache/6d/7fce9fee471194aa8b5b6e47267f03' to 'ssh://[email protected]/home/mroutis/cache/6d/7fce9fee471194aa8b5b6e47267f03' - ChannelException(1, 'Administratively prohibited')

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
[##############################] 100% data/2.txt
[##############################] 100% data
[##############################] 100% data/1.txt
[##############################] 100% data/2.txt
[##############################] 100% data
[##############################] 100% data/1.txt

ERROR: failed to push data to the cloud - 1 file(s) failed to upload

Having any troubles?. Hit us up at https://dvc.org/support, we are always happy to help!
Exception TypeError: 'super() argument 1 must be type, not None' in <bound method SSHPool.__del__ of <dvc.remote.ssh.pool.SSHPool object at 0x7f9b346f2050>> ignored

Then, if I do dvc push -r ssh again to push the remaining file (data/3.txt):

Preparing to upload data to 'ssh://[email protected]/home/mroutis/cache'
Preparing to collect status from ssh://[email protected]/home/mroutis/cache
Collecting information from local cache...
[##############################] 100%

Collecting information from remote cache...
[##############################] 100%
[##############################] 100% Analysing status
[##############################] 100% data/3.txt
[##############################] 100% data/3.txt
Exception Unhandled exception in thread started by
Traceback (most recent call last):
TypeError: 'super(  File "/usr/lib64/python2.7/threading.py", line 774, in __bootstrap
) argument 1 must be type, not None'    self.__bootstrap_inner()
  File "/usr/lib64/python2.7/threading.py", line 814, in __bootstrap_inner
     in <bound method SSHPool.__del__ of <dvc.remote.ssh.pool.SSHPool object at 0x7fc043a70710>> ignored
(self.name, _format_exc()))
  File "/usr/lib64/python2.7/traceback.py", line 241, in format_exc
    etype, value, tb = sys.exc_info()
AttributeError: 'NoneType' object has no attribute 'exc_info'

[Suor]

The original users issue:

DEBUG: SELECT count from state_info WHERE rowid=?                                                                                                
DEBUG: fetched: [(217270,)]                                                                                                                      
DEBUG: UPDATE state_info SET count = ? WHERE rowid = ?                                                                                           
ERROR: unexpected error - 'NoneType' object is not iterable                                                                                      
------------------------------------------------------------                                                                                     
Traceback (most recent call last):                                                                                                               
  File "/opt/miniconda2/lib/python2.7/site-packages/dvc/main.py", line 39, in main                                                       
    ret = cmd.run_cmd()                                                                                                                          
  File "/opt/miniconda2/lib/python2.7/site-packages/dvc/command/base.py", line 63, in run_cmd                                            
    return self.run()                                                                                                                            
  File "/opt/miniconda2/lib/python2.7/site-packages/dvc/command/data_sync.py", line 21, in run                                           
    return self.do_run()                                                                                                                         
  File "/opt/miniconda2/lib/python2.7/site-packages/dvc/command/data_sync.py", line 67, in do_run                                        
    recursive=self.args.recursive,                                                                                                               
  File "/opt/miniconda2/lib/python2.7/site-packages/dvc/repo/push.py", line 27, in push                                                  
    used, jobs, remote=remote, show_checksums=show_checksums                                                                                     
  File "/opt/miniconda2/lib/python2.7/site-packages/dvc/data_cloud.py", line 124, in push                                                
    show_checksums=show_checksums,                                                                                                               
  File "/opt/miniconda2/lib/python2.7/site-packages/dvc/remote/local/__init__.py", line 427, in push                                     
    download=False,                                                                                                                              
  File "/opt/miniconda2/lib/python2.7/site-packages/dvc/remote/local/__init__.py", line 399, in _process                                 
    download=download,                                                                                                                           
  File "/opt/miniconda2/lib/python2.7/site-packages/dvc/remote/local/__init__.py", line 327, in status                                   
    remote_exists = list(remote.cache_exists(md5s))                                                                                              
  File "/opt/miniconda2/lib/python2.7/site-packages/dvc/remote/base.py", line 678, in cache_exists                                       
    ret = list(itertools.compress(checksums, in_remote))                                                                                         
TypeError: 'NoneType' object is not iterable                                                                                                     
------------------------------------------------------------

[efiop]

Looks like 4 issues. Need to investigate.

[darabi]

The underlying problem is an error opening the sftp connection which is clearly visible when I use paramiko directly:

Traceback (most recent call last):
  File "sftp_stat.py", line 24, in <module>
    sftp = paramiko.SFTPClient.from_transport(transport)
  File "/opt/python-env/python3/lib/python3.6/site-packages/paramiko/sftp_client.py", line 169, in from_transport
    chan.invoke_subsystem("sftp")
  File "/opt/python-env/python3/lib/python3.6/site-packages/paramiko/channel.py", line 72, in _check
    return func(self, *args, **kwds)
  File "/opt/python-env/python3/lib/python3.6/site-packages/paramiko/channel.py", line 283, in invoke_subsystem
    self._wait_for_event()
  File "/opt/python-env/python3/lib/python3.6/site-packages/paramiko/channel.py", line 1226, in _wait_for_event
    raise e
paramiko.ssh_exception.SSHException: Channel closed.

It would be nice to have all exceptions surface at the command line and of course batch_exists should not continue in case of an sftp error.

With that SSHException right in front of me, I would have realised that my server is not configured for sftp (I implicitly assumed that dvc would use scp for file transfer).

[Suor]

@darabi so did it work after sftp was set up?

[Suor]

Reproduced the original issue by turning off SFTP:

Traceback (most recent call last):
  File "/home/suor/projects/dvc/dvc/main.py", line 41, in main
    ret = cmd.run_cmd()
  File "/home/suor/projects/dvc/dvc/command/base.py", line 63, in run_cmd
    return self.run()
  File "/home/suor/projects/dvc/dvc/command/data_sync.py", line 21, in run
    return self.do_run()
  File "/home/suor/projects/dvc/dvc/command/data_sync.py", line 67, in do_run
    recursive=self.args.recursive,
  File "/home/suor/projects/dvc/dvc/repo/push.py", line 27, in push
    used, jobs, remote=remote, show_checksums=show_checksums
  File "/home/suor/projects/dvc/dvc/data_cloud.py", line 124, in push
    show_checksums=show_checksums,
  File "/home/suor/projects/dvc/dvc/remote/local/__init__.py", line 427, in push
    download=False,
  File "/home/suor/projects/dvc/dvc/remote/local/__init__.py", line 399, in _process
    download=download,
  File "/home/suor/projects/dvc/dvc/remote/local/__init__.py", line 327, in status
    remote_exists = list(remote.cache_exists(md5s))
  File "/home/suor/projects/dvc/dvc/remote/base.py", line 678, in cache_exists
    ret = list(itertools.compress(checksums, in_remote))
TypeError: 'NoneType' object is not iterable

[Suor]

Since confusing messages and error hiding is resolved in #2288 and the original issue is a misconfigured SSH, I close it in favor or #2292.