Dvc: using ssh as a remote, unexpected error - Channel closed

[aabramovrepo]

Hi there,

I'm facing the following issue while using ssh as a remote for storing data on a disk station manager by Synology. Here is the error I'm getting after dvc push:

ERROR: unexpected error - Channel closed.

The same remote configuration works just fine when using a standard Desktop machine with Ubuntu 18 as a server. The DVC remote was created as follows:

dvc remote add remote-ssh ssh://<user name>@<ip address>:<path on the server>

dvc config core.remote remote-ssh

dvc remote modify remote-ssh ask_password true

Does anybody have an idea what needs to be changed in the ssh config?

Thanks in advance for any advice!

Best,
Alexey

[aabramovrepo]

Here is SFTP subsystem configuration from /etc/ssh/sshd_config:

Subsystem sftp internal-sftp -f DAEMON -u 000

[efiop]

Hi @aabramovrepo !

Could you show full log for dvc push -v, please? Also, are you able to ssh into that machine yourself?

[efiop]

@aabramovrepo Also, are you able to sftp into that machine from CLI?

[aabramovrepo]

Hi, thanks for the quick response!

Could you show full log for dvc push -v, please? Also, are you able to ssh into that machine yourself?

DEBUG: Establishing ssh connection with '<ip address>' through port '22' as user '<user name>'              
DEBUG: Establishing ssh connection with '<ip address>' through port '22' as user '<user name>'              
DEBUG: Establishing ssh connection with '<ip address>' through port '22' as user '<user name>'              
DEBUG: Establishing ssh connection with '<ip address>' through port '22' as user <user name>'              
DEBUG: SELECT count from state_info WHERE rowid=?                                                       
DEBUG: fetched: [(15275,)]
DEBUG: UPDATE state_info SET count = ? WHERE rowid = ?
ERROR: unexpected error - Channel closed.
------------------------------------------------------------
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/dvc/main.py", line 50, in main
    ret = cmd.run()
  File "/usr/local/lib/python3.6/dist-packages/dvc/command/data_sync.py", line 51, in run
    recursive=self.args.recursive,
  File "/usr/local/lib/python3.6/dist-packages/dvc/repo/__init__.py", line 35, in wrapper
    ret = f(repo, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/dvc/repo/push.py", line 27, in push
    return self.cloud.push(used, jobs, remote=remote)
  File "/usr/local/lib/python3.6/dist-packages/dvc/data_cloud.py", line 63, in push
    show_checksums=show_checksums,
  File "/usr/local/lib/python3.6/dist-packages/dvc/remote/local.py", line 378, in push
    download=False,
  File "/usr/local/lib/python3.6/dist-packages/dvc/remote/local.py", line 351, in _process
    download=download,
  File "/usr/local/lib/python3.6/dist-packages/dvc/remote/local.py", line 272, in status
    md5s, jobs=jobs, name=str(remote.path_info)
  File "/usr/local/lib/python3.6/dist-packages/dvc/remote/ssh/__init__.py", line 333, in cache_exists
    ret = list(itertools.compress(checksums, in_remote))
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 586, in result_iterator
    yield fs.pop().result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 432, in result
    return self.__get_result()
  File "/usr/lib/python3.6/concurrent/futures/_base.py", line 384, in __get_result
    raise self._exception
  File "/usr/lib/python3.6/concurrent/futures/thread.py", line 56, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/local/lib/python3.6/dist-packages/dvc/remote/ssh/__init__.py", line 326, in exists_with_progress
    return self.batch_exists(chunks, callback=pbar.update_desc)
  File "/usr/local/lib/python3.6/dist-packages/dvc/remote/ssh/__init__.py", line 294, in batch_exists
    channels = ssh.open_max_sftp_channels()
  File "/usr/local/lib/python3.6/dist-packages/dvc/remote/ssh/connection.py", line 306, in open_max_sftp_channels
    self._sftp_channels.append(self._ssh.open_sftp())
  File "/usr/local/lib/python3.6/dist-packages/paramiko/client.py", line 556, in open_sftp
    return self._transport.open_sftp_client()
  File "/usr/local/lib/python3.6/dist-packages/paramiko/transport.py", line 1097, in open_sftp_client
    return SFTPClient.from_transport(self)
  File "/usr/local/lib/python3.6/dist-packages/paramiko/sftp_client.py", line 169, in from_transport
    chan.invoke_subsystem("sftp")
  File "/usr/local/lib/python3.6/dist-packages/paramiko/channel.py", line 72, in _check
    return func(self, *args, **kwds)
  File "/usr/local/lib/python3.6/dist-packages/paramiko/channel.py", line 283, in invoke_subsystem
    self._wait_for_event()
  File "/usr/local/lib/python3.6/dist-packages/paramiko/channel.py", line 1226, in _wait_for_event
    raise e
paramiko.ssh_exception.SSHException: Channel closed.
------------------------------------------------------------

yes, I can ssh into the server machine by myself.

[aabramovrepo]

@aabramovrepo Also, are you able to sftp into that machine from CLI?

Nope! I'm getting the following error message:

subsystem request failed on channel 0
Connection closed

[efiop]

@aabramovrepo Oh, if you are not able to sftp into that machine without dvc, then I would start with sorting out that issue first. I'm not that familiar with sftp configuration and synology so I don't really know if that is a known issue or not. Googling around for synology subsystem request failed on channel 0 I'm finding plenty of links like https://community.synology.com/enu/forum/17/post/115230 discussing seemingly the same issue. Please check out those links, see if anything works for you. I'll close this issue for now, as it is a synology issue. Please ping us and reopen this issue if sftp from cli starts working, but dvc still doesn't work. Thanks! :slightly_smiling_face:

[shcheklein]

@efiop should we update our docs (and website?) to mention that SFTP support is required?

[efiop]

@shcheklein It seems pretty obvious, but sure https://github.com/iterative/dvc.org/issues/781

[shcheklein]

@efiop may be I don't quite understand this (like is SFTP the only way to deal with SSH?), could you elaborate a little bit why is it obvious? May be we don't need that ticket indeed.

[efiop]

@shcheklein Not aware of any other way. But we don't mention SFTP in our docs at all, so I guess it is worth documenting.

[aabramovrepo]

@efiop I have resolved the problem with sftp, now I can ssh and sftp into the disk station machine, furthermore, I can copy files and directories there via scp. However, dcp push gives me the following error message:

ERROR: failed to upload '.dvc/cache/94/d190064798aeae79ea2a1eafd0042e' to 'ssh://gituser@<ip address>/volume1/DVC/94/d190064798aeae79ea2a1eafd0042e' - unable to create remote directory '/volume1': [Errno 13] Permission denied

Do you know what might be the reason for that? I'm a bit surprised seeing no colon between ip address and directory on the server. dvc remote list prints the following:

remote-ssh-nas ssh://gituser@<ip address>:/volume1/DVC

for copying files via scp the following command is used (with colon):

scp -r .dvc/cache/ gituser@<ip address>:/volume1/DVC

and it works just fine.

Thanks in advance for your help!

Cheers,
Alexey

[efiop]

@aabramovrepo Glad to hear you've got ssh and sftp figured out!

Regarding the colon, it doesn't matter, both formats are valid, it is just how we show it in logs.

Regarding the "Permission denied", I suppose /volume1 exists already, right? If so, I would suspect that your sftp root is not in /, but somewhere else, most likely. Similar to https://github.com/iterative/dvc/issues/2535 . Could you sftp into that machine and run ls to ensure that it is indeed the directory that you expect sftp to be in(e.g. you should see volume1 directory)?

[aabramovrepo]

@efiop thanks for the instant respond!

yeah, volume1 exists already, that's correct. After sftp into the server machine I'm landing in voulme1 so ls shows me the target DVC directory. Any ideas?

[efiop]

@aabramovrepo Yep, so your sftp server is configured to have its root in /volume1 instead of /(talking about physical root). So you need to adjust your remote url to be relative to /volume1 and not /. So it should be ssh://gituser@<ip address>:/DVC. Give it a try and let us know if that works 🙂

[efiop]

@aabramovrepo Btw, when you ssh(not sftp) into your machine and run ls -la /, do you see a physical root (volume1 dir is present in the output) or/volume1?

[aabramovrepo]

@efiop yeah, it helped, dvc push works now, thanks a lot man!

To your second question: ls -la / on the server machine (getting there via ssh) lists the physical root, so volume1 is present there.

[efiop]

@aabramovrepo Sure! Glad to hear it works now! 🙂

Thanks for the info! Makes me think that we might be able to use that to detect that we are not in the physical root and print a meaningful hint or something https://github.com/iterative/dvc/issues/2812. If not, we'll at least document it https://github.com/iterative/dvc.org/issues/802. Thanks for the feedback!

[wbolster]

have resolved the problem with sftp, now I can ssh and sftp into the disk station machine

hey @aabramovrepo, it seems i'm having a similar issue. ssh works fine, but sftp does not, resulting in subsystem request failed on channel 0, while the config (both synology DSM UI and sshd_config) seem to indicate SFTP is enabled.

care to share how you solved your issue? it may help me.

_update: fixed_. it seems that somehow the permission for my user to use the ‘FTP’ application (also applicable to sftp apparently) was disabled. i reenabled via the synology DSM control panel by editing my user account.