Vscode-remote-release: Could not establish SSH connection: flock: 99: Input/output error

[arglog]

• VS Code Version: 1.36.0-insider
• Remote - SSH version: 0.42.2

I was connecting to a remote machine using SSH-Remote. I got a pop-up window showing Could not establish connection to "xxxx.yyyy.zzz".

The error message in the output panel is

[email protected]
SSH Resolver called for "ssh-remote+xxxx.yyyy.zzz"
SSH Resolver called for host: xxxx.yyyy.zzz
Setting up SSH remote "xxxx.yyyy.zzz"
Using commit id "b7ca272b6e041e42f9edfe48f4d559c2044cdaae" and quality "insider" for server
Install and start server if needed
> 
> ]0;C:\WINDOWS\SYSTEM32\cmd.exe
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> flock: 
> flock: 99: Input/output error
> Installation already in progress...       
> 39db3155-22cc-4505-b432-8698d5e68ff9##24##
> flock: 99: Input/output error
> 
"install" terminal command done
Received install output: flock: 99: Input/output error
Failed to parse remote port from server output: flock: 99: Input/output error

[dawnmy]

The same problem

[saeedizadi]

The same problem here

[roblourens]

Are you all using the NFS file system?

[arglog]

Hi @roblourens , yes I'm using the NFS file system.

Could @dawnmy and @saeedizadi also provide the information please?

[saeedizadi]

Yes, I'm using NFS file system as well.

[saeedizadi]

Any update on. this? I'm suffering this issue so much. Any workaround?

[roblourens]

Sorry, I have to find time to set up an NFS drive and do some experimentation. Not clear if I have to find another locking solution entirely or what. I think I read that flock should be supported on NFSv4. Do you know which NFS you are using?

Our file locking is based on https://gist.github.com/przemoc/571091 if you care to do any experimentation yourself.

[arglog]

Hi @roblourens
I'm happy to help out. Can you provide more information about

1. how to check which NFS I'm using?
2. how to do the experiment using the code in the Gist?

[roblourens]

1. I have no idea
2. You could try running that script, and replace the /var/lock/ path with a path that is inside your user directory on the NFS drive

But maybe the lockfiles should just be stored in /var/lock. I assume that /var/lock is not an NFS drive for you correct?

[arglog]

@roblourens

1. I'm using NFS4.
2. What the exact command should I use? I tried bash lockable_script_boilerplate.sh but got nothing in the output.

[roblourens]

Well you should not get this input/output error.

You could set up a situatiion where one instance of the script acquires a lock and sleeps for a bit then a second instance of the script tries to acquire the lock and fails nicely with no weird error messages.

[roblourens]

So I set up an NFS drive for testing and I am able to use vscode remote on that drive with no issues 🤷‍♂

Can you run mount and share the line for the NFS drive? Mine looks like

127.0.0.1:/datadrive/shared on /mnt/shared type nfs4 (rw,relatime,vers=4.2,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1)

Also the output of uname -a?

And if you know the name of the command to mount the nfs drive, that would be helpful as well.

[arglog]

@roblourens

~$ mount
hamms-0.xxxx.yyy:/data01/xxxxxx/xxxxx/users/yyyyy on /cellar/xxxx/yyyyy type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=169.228.36.70,local_lock=none,addr=169.228.36.103)

~$ uname -a
Linux ortugo 4.4.0-87-generic #110-Ubuntu SMP Tue Jul 18 12:55:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

[pmgurman]

Can add that I am seeing this error on an Ubuntu 14.04.5 LTS server, which is mounting the home directory using nfs version 4.0. It works with a Centos 7.6 server using nfs version 4.1 with the same home directory.

[roblourens]

I was able to mount with version 4.0 and it still works.

127.0.0.1:/datadrive/shared on /mnt/shared type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=127.0.0.1,local_lock=none,addr=127.0.0.1)

I wonder whether it works because I am technically sharing to the same machine, maybe it takes some shortcut when it knows the drive is really local.

[roblourens]

Can someone tell me the output of dpkg -l | grep nfs-common so I can use exactly the same version of the client code?

[roblourens]

And maybe dpkg -l | grep nfs-kernel-server if you have access to the server

[arglog]

@roblourens

~$ dpkg -l | grep nfs-common
ii  nfs-common                         1:1.2.8-9ubuntu12.1                        amd64        NFS support files common to client and server

There is no output for dpkg -l | grep nfs-kernel-server.

[roblourens]

Now I am trying on 14.04 with 1:1.2.8-6ubuntu1.2 and it still seems fine. However it takes an extremely long time to untar the downloaded package...

For dpkg -l | grep nfs-kernel-server I meant run that on the server, if you have access to it.

[roblourens]

And here is a test script is you can try running against your NFS $HOME. You will probably get the same error, I just want to verify that the simplest possible case produces the same result.

LOCKFILE=$HOME/test.lock
LOCKFD=99

# PRIVATE
_lock()             { flock -$1 $LOCKFD; }
_no_more_locking()  { _lock u; _lock xn && rm -f $LOCKFILE; }
_prepare_locking()  { eval "exec $LOCKFD>\"$LOCKFILE\""; trap _no_more_locking EXIT; }

# ON START
echo 0
_prepare_locking

# PUBLIC
exlock_now()        { _lock xn; }  # obtain an exclusive lock immediately or fail
exlock()            { _lock x; }   # obtain an exclusive lock
shlock()            { _lock s; }   # obtain a shared lock
unlock()            { _lock u; }   # drop a lock

### BEGIN OF SCRIPT ###

# Simplest example is avoiding running multiple instances of script.

echo 1
exlock_now || exit 1
echo "exlock_now exited with $?"
sleep 10
echo 2
unlock
echo 3

[pmgurman]

Can confirm that above script causes the original error on the 'exlock_now || exit 1' line (and severs my ssh connection).

I get the following from dpkg -l | grep nfs-common
ii nfs-common 1:1.2.8-6ubuntu1.2 amd64 NFS support files common to client and server

In my case, I don't believe its the nfs host, because I can use the extension to go to a different computer that mounts the same home directory.

The mount command shows me the following about the mount

(rw,sec=sys,addr=x.x.x.x,clientaddr=0.0.0.0)

[roblourens]

Same OS, same nfs-common version, same mount details.

Well I can add a setting to put the lock in /tmp. Originally locks were in /var/lock but on some systems users do not have write access to /var/lock. Would that work? You could change the path in the script to somewhere in /tmp to test.

One downside is that this would not be safe when the home folder is mapped to two different machines, and vscode connects to those two machines simultaneously. We can probably live with that.

[arglog]

@roblourens
Yes. I was running dpkg -l | grep nfs-kernel-server on the remote machine, not on my local laptop. But there is no output.

For the test script, here is the output

0
1
exlock_now exited with 0
2
3

[roblourens]

I mean, the "server" in this case is the file server that hosts your NFS directory. This probably is not the "remote" machine that you are connecting to, that is the NFS client.

Did you run the test script on the remote machine? Or your local laptop. It should be run on the remote where the NFS drive is.

[arglog]

Oh, I see. I don't have the access to the server hosting the NFS directory.

Yes, I ran the test script on the remote machine.

[roblourens]

Well then 🤷‍♂

Could you try another thing for me?

• Uninstall the ssh extension
• Download this, change extension .vsix remote-ssh-0.44.1.zip
• Install it (e.g. with "Install from VSIX" command)
• Try again, this will keep the lock in /tmp, let me know whether it works

[pmgurman]

That seems to work.

It was hard to test though because it kept updating itself before I could test it and I kept forgetting that there were hanging processes on the remote from previous attempts.

[arglog]

@roblourens
It works for the first time. I was able to connect to the remote server. However, after the connection, it failed to open a folder on the server, showing the same error (flock: 99 Input/output error). If I disconnect from the server and re-connect again, I got the same error. Only after uninstalling and re-installing the extension can I successfully connect to the server again.

@pmgurman Were you able to open folders on the remote machine and successfully re-connect to the remote for multiple times?

[arglog]

@pmgurman How did you prevent the extension keeping updating itself? I found it updates to 0.44.2 after I connecting to the server. Maybe this is the reason that I was not able to open a folder after the connection was established.

[pmgurman]

@arglog I was just about to say the same. I guess it is updating itself on restart of VSCode, and updating from the internet and not the modified package version

[roblourens]

Sorry I should have bumped the version number to prevent that. You can disable extensions.autoUpdate temporarily to prevent it from overwriting the vsix.

[arglog]

@roblourens Thanks. This works for me. Using this VSIX extension I successfully connected to the remote machine and opened project folders. I also reproduced the connection several times and there was no error.

[roblourens]

Tomorrow's nightly build of the ssh extension will have the setting remote.SSH.lockfilesInTmp that you can set to do the same thing.

[pmgurman]

@roblourens Forgive my ignorance, but where would I find the nightly?

[roblourens]

https://marketplace.visualstudio.com/items?itemName=ms-vscode-remote.remote-ssh-nightly

Make sure you uninstall the stable version before using the nightly version

[csaar95]

@roblourens nightly version for me also working, thanks.

[pmgurman]

@roblourens One side effect I am seeing of remote.SSH.lockfilesInTmp is that when I start a terminal on the remote machine it does not load my bash settings from my home folder, i.e. I have a .bash_profile file that is normally read. Do you have any ideas what is happening? I didn't know if I should post this here or in a new issue, so apologies if it is in the wrong spot?

[roblourens]

That shouldn't be related to this. You can open a new issue with some details.

[yosider]

It worked after installing nightly version of remote SSH extension and checking "Lockfiles In Tmp". Thanks!