Git: [2.23.0] bash: /dev/fd/62: No such file or directory

[JoseGalRe]

• [x] I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

git version 2.23.0.windows.1
cpu: x86_64
built from commit: 4db2e5cc9e1522131a039cbad3970f147a39f0ce
sizeof-long: 4
sizeof-size_t: 8

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

Microsoft Windows [Version 10.0.17763.678]

• What options did you set as part of the installation? Or did you choose the
  defaults?

Editor Option: Notepad++
Custom Editor Path:
Path Option: Cmd
SSH Option: OpenSSH
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Enabled
Enable Builtin Interactive Add: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

git-bash.exe

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

cat < <(echo "hello")

• What did you expect to occur after running these commands?

• What actually happened instead?

bash: /dev/fd/62: No such file or directory

Note: on '2.22.0' works fine

[PhilipOakley]

In a dumb, just copy the commands, mode, I was _able to repeat the exercise_.

I then tried:
phili@Philip-Win10 MINGW64 ~ $ <(echo "hello") bash: /dev/fd/63: No such file or directory
(the fd number is one higher, 63 rather than 62.)
ASCII 62 is 3E ">" if that has any relevance.

I suspect that this because of the Cygwin update in the release notes.

However, as someone who is not a happy scripter, I wasn't sure what the command hoped to do anyway. Perhaps you could describe your expectation and decompose what you hoped it would do?

[dscho]

Can you open Git Bash as administrator, call mkdir -p /dev/fd, and then try again (not as admin)?

[PhilipOakley]

@dscho On my machine (git opened as admin):

phili@Philip-Win10 MINGW64 ~
$  mkdir -p /dev/fd
mkdir: cannot create directory ‘/dev/fd’: Read-only file system

phili@Philip-Win10 MINGW64 ~
$ git version --build-options
git version 2.23.0.windows.1
cpu: x86_64
built from commit: 4db2e5cc9e1522131a039cbad3970f147a39f0ce
sizeof-long: 4
sizeof-size_t: 8

phili@Philip-Win10 MINGW64 ~
$

Just a data point...

[JoseGalRe]

In a dumb, just copy the commands, mode, I was _able to repeat the exercise_.

I then tried:

$ <(echo "hello")
bash: /dev/fd/63: No such file or directory

(the fd number is one higher, 63 rather than 62.)
ASCII 62 is 3E ">" if that has any relevance.

I suspect that this because of the Cygwin update in the release notes.

However, as someone who is not a happy scripter, I wasn't sure what the command hoped to do anyway. Perhaps you could describe your expectation and decompose what you hoped it would do?

of course the command with "cat" is just an example, I use "< <(..)" usually to insert elements in an array, but "cat" is a simple and short example.

yes, this breaks all my scripts, where I use "< <(..)"
I have read the release notes and I also think it is due to the Cygwin update.

Can you open Git Bash as administrator, call mkdir -p /dev/fd, and then try again (not as admin)?

whether admin or not, it is in read-only mode

I tried:

mkdir -p /dev/fd
ln -s /proc/self/fd /dev/fd

but the answer is always the same:

read-only file system

[dscho]

Okay, this is what I did, inspired by https://sourceware.org/ml/cygwin/2009-02/msg00389.html:

1. as administrator, in a PowerShell, I called mkdir C:\Program Files\Git\dev.
2. as administrator, in a Git Bash, I called ln -s /proc/self/fd /dev/fd

After that, things seemed to work for me.

Note: it is important to create the directory _outside_ of Git Bash, as the MSYS2 runtime pretends that that directory exists even if it doesn't. And the ln call fails unless the directory _really_ exists, so it needs to be created explicitly.

Can you try that? If that fixes the problems, maybe you would be interested in working on a patch to install.iss to fix this for good?

[stepro]

This has broken many of my scripts as well. Seems there should be a bash postinstall script like this that should run to automatically pre-create these device links. What changed between 2.22 and 2.23?

[JoseGalRe]

Okay, this is what I did, inspired by https://sourceware.org/ml/cygwin/2009-02/msg00389.html:

1. as administrator, in a PowerShell, I called mkdir C:\Program Files\Git\dev.
2. as administrator, in a Git Bash, I called ln -s /proc/self/fd /dev/fd

After that, things seemed to work for me.

Note: it is important to create the directory _outside_ of Git Bash, as the MSYS2 runtime pretends that that directory exists even if it doesn't. And the ln call fails unless the directory _really_ exists, so it needs to be created explicitly.

Thanks mate, now works perfectly!
maybe in the next releases the installer should create the dev dir and make the symbolic link? xD

Can you try that? If that fixes the problems, maybe you would be interested in working on a patch to install.iss to fix this for good?

I don't know very well how git-bash could run as an administrator on inno setup during installation to run a postinstall as stepro says

[PhilipOakley]

1. Opened PowerShell as administrator, I called mkdir "C:\Program Files\Git\dev" (with quotes).
2. Opened a Git Bash as administrator, I called ln -s /proc/self/fd /dev/fd

Also worked for me [my data point]. I.e.

phili@Philip-Win10 MINGW64 /c/git-sdk-64/usr/src/git ((a85708cd88...))
$ cat < <(echo "hello")
hello

Earlier, in a regular (non-admin) bash shell (V2.23.0) I'd seen, (by chance, I'd forgotten to cd back to my repo)

phili@Philip-Win10 MINGW64 /c/git-sdk-64/usr/src/git ((a85708cd88...))
$ cd /dev/

phili@Philip-Win10 MINGW64 /dev
$ ls
clipboard  console  null  random  sda1  sda4  sdb2  ttyS0    zero
conin      dsp      ptmx  scd0    sda2  sdb   sr0   urandom
conout     full     pty0  sda     sda3  sdb1  tty   windows

phili@Philip-Win10 MINGW64 /dev
$ git help config
Error: Current working directory is a virtual Cygwin directory which does
not exist for a native Windows application.
Can't start native Windows application from here.

which at least confirms the underlying Cygwin aspect of the virtual file system at that point.

[stepro]

I don't know very well how git-bash could run as an administrator on inno setup during installation to run a postinstall as stepro says

Clearly some part of the 2.22 installation used to do this. I reverted back and my /dev directory looks like this:

drwxr-xr-x 1 stepro 1049089        0 Aug 19 16:16 ./
drwxr-xr-x 1 stepro 1049089        0 Aug 19 15:23 ../
crw-rw-rw- 1 stepro 1049089  13, 254 Nov 30  2006 clipboard
crw-rw-rw- 1 stepro 1049089   5, 255 Aug 20 08:56 conin
crw-rw-rw- 1 stepro 1049089   5, 254 Aug 20 08:56 conout
crw-rw-rw- 1 stepro 1049089   5,   1 Aug 20 08:56 console
crw-rw-rw- 1 stepro 1049089  14,   3 Aug 20 08:56 dsp
lrwxrwxrwx 1 stepro 1049089       13 Aug 19 14:08 fd -> /proc/self/fd/
crw-rw-rw- 1 stepro 1049089   1,   7 Aug 20 08:56 full
drwxr-xr-x 1 stepro 1049089        0 Aug 19 14:08 mqueue/
crw-rw-rw- 1 stepro 1049089   1,   3 Aug 20 08:56 null
crw-rw-rw- 1 stepro 1049089   5,   2 Aug 20 08:56 ptmx
crw------- 1 stepro 1049089 136,   0 Aug 20 08:56 pty0
crw-rw-rw- 1 stepro 1049089   1,   8 Aug 20 08:56 random
brw-rw-rw- 1 stepro 1049089   8,   0 Aug 20 08:56 sda
brw-rw-rw- 1 stepro 1049089   8,   1 Aug 20 08:56 sda1
brw-rw-rw- 1 stepro 1049089   8,   2 Aug 20 08:56 sda2
brw-rw-rw- 1 stepro 1049089   8,   3 Aug 20 08:56 sda3
brw-rw-rw- 1 stepro 1049089   8,   4 Aug 20 08:56 sda4
brw-rw-rw- 1 stepro 1049089   8,  16 Aug 20 08:56 sdb
brw-rw-rw- 1 stepro 1049089   8,  17 Aug 20 08:56 sdb1
brw-rw-rw- 1 stepro 1049089   8,  18 Aug 20 08:56 sdb2
drwxr-xr-x 1 stepro 1049089        0 Aug 19 14:08 shm/
lrwxrwxrwx 1 stepro 1049089       15 Aug 19 14:08 stderr -> /proc/self/fd/2
lrwxrwxrwx 1 stepro 1049089       15 Aug 19 14:08 stdin -> /proc/self/fd/0
lrwxrwxrwx 1 stepro 1049089       15 Aug 19 14:08 stdout -> /proc/self/fd/1
crw-rw-rw- 1 stepro 1049089   5,   0 Aug 20 08:56 tty
crw-rw-rw- 1 stepro 1049089   1,   9 Aug 20 08:56 urandom
crw-rw-rw- 1 stepro 1049089  13, 255 Aug 20 08:56 windows
crw-rw-rw- 1 stepro 1049089   1,   5 Aug 20 08:56 zero

In the new installation, there are no symbolic links.

Workarounds are great, but this needs to be addressed at the source. Fundamental aspects of Bash assume the existence of these links, as pointed out here, e.g.:

# Install /dev/fd, /dev/std{in,out,err}.  The bash builtin test was compiled
# to assume these exist, so use /bin/test to really check.

[dscho]

The solution will be to have the InnoSetup-based installer create this directory before the post-install script runs (I already pointed out why that shell script cannot create it).

I trust you folks to make this happen, in a PR to the build-extra repository, changing install.iss accordingly.

[stepro]

This doesn't fix the root cause, since install.iss never created this directory before - something else did. And whatever that is/was (MSYS2, Cygwin, something else? I don't understand all these dependencies) is also now broken. Shouldn't we understand the root of the problem before patching elsewhere?

[dakotahawkins]

The latest installs of cygwin and msys2 seem fine, /dev/fd -> /proc/self/fd.

[dscho]

Shouldn't we understand the root of the problem before patching elsewhere?

@stepro go to town, then! For starters, you might want to unpack portable versions of the working vs non-working Git for Windows, then investigate the differences in the /etc/package-versions.txt file, and based on that, replace files in the non-working versions until things work.

You will want to unpack the portable Git's via 7-Zip, though, and then prevent the /etc/post-install/99-post-install-cleanup.post script from removing the post-install scripts (by deleting it before running git-bash.exe). I see, for example, that the /dev/fd link itself is created in /etc/post-install/01-devices.post.

[stepro]

Thanks @dscho, these links are very helpful. I'll take a look.

[stepro]

Ok, the root cause comes from this change, which alters the behavior of mkdir such that it will no longer create an entry in the Windows filesystem for Cygwin-only directories like /dev. This commit was introduced in Git for Windows as part of the upgrade to by moving to msys2-runtime v3.0.7 patch 3.

Previously, mkdir succeeded, allowing this line to both create /dev and successfully add files like /dev/fd to it. Now, it fails to create the directory with the error mkdir: cannot create directory '/dev': File exists (although this error is not seen because it pipes stderr to /dev/null), and the script proceeds because /dev does effectively exist and passes the existence check. However, by not being an actual directory in the Windows filesystem, Cygwin doesn't allow creation of any files in that directory, and returns a "Read-only file system" error. Therefore creation of the shm and mqueue directories fail, and adding the various /dev/fd links also fails. When this is called by post-install.bat, the output is:

"running post-install"
mkdir: cannot create directory '/dev/shm': Read-only file system

Creating /dev/shm directory failed.
POSIX semaphores and POSIX shared memory will not work

mkdir: cannot create directory '/dev/mqueue': Read-only file system

Creating /dev/mqueue directory failed.
POSIX message queues will not work

ln: failed to create symbolic link '/dev/stdin': Read-only file system
ln: failed to create symbolic link '/dev/stdout': Read-only file system
ln: failed to create symbolic link '/dev/stderr': Read-only file system
ln: failed to create symbolic link '/dev/fd': Read-only file system
'C:\windows\system32\drivers\etc\hosts' -> '/etc/hosts'
'C:\windows\system32\drivers\etc\protocol' -> '/etc/protocols'
'C:\windows\system32\drivers\etc\services' -> '/etc/services'
'C:\windows\system32\drivers\etc\networks' -> '/etc/networks'

Because this script has no error handling, the Git for Windows installer thinks that post-install.bat succeeded when really it should have failed.

Why is this not failing in MSYS2 itself? Because the current installer is from May 24 and the patch wasn't committed to Cygwin until June 5.

Why is this not failing in Cygwin installs? Same deal - the latest Cygwin installer puts down a cygwin1.dll (where the compiled code that was patched lives) from April 30.

So, this problem is likely to start showing up in MSYS2 and Cygwin if they don't do something about it. Who can be contacted about ensuring those projects don't break?

Git for Windows seems ahead of the game by shipping a newer version of Cygwin than Cygwin itself that already manifests the issue. If folks from Cygwin agree that the right fix for this is for a given installer to manually create the directory beforehand, then that's what should be done. Otherwise, it could be added to the Git for Windows installer as a temporary workaround until a new version of Cygwin is picked up that properly accounts for it.

[dscho]

@stepro _excellent_ analysis, I am impressed!

Who can be contacted about ensuring those projects don't break?

For Cygwin, I would suggest following https://cygwin.com/problems.html, for MSYS2 I would suggest opening a ticket in https://github.com/msys2/MSYS2-packages.

If folks from Cygwin agree that the right fix for this is for a given installer to manually create the directory beforehand, then that's what should be done. Otherwise, it could be added to the Git for Windows installer as a temporary workaround until a new version of Cygwin is picked up that properly accounts for it.

I actually think that it might make most sense for the installers/archives to already include that directory.

Originally, I thought make-file-list.sh (which is used by the scripts that generate the Git for Windows installer, portable Git, MinGit, etc) would be the right place, but I am worried that it would cause at least one of the consumers might mistake this for a sign that it should add that directory _with contents_, and we don't want that, now, do we ;-)

[dakotahawkins]

Would it be prudent to have a patch revert that change for this project in the meantime (while following up or waiting on upstream projects)?

[dscho]

The commit message of that patch suggests that it fixes a real problem, though... So I'd vote for not reverting it ;-)

[dakotahawkins]

Yeah that's fair. I'm uncertain what was actually happening when we were allowed to clobber-create it before and how exactly it avoided this:

However, by not being an actual directory in the Windows filesystem, Cygwin doesn't allow creation of any files in that directory, and returns a "Read-only file system" error.

[WayneD]

Is the following a valid work-around? Changing this line in the cited 01-devices.post shell script:

local DEVDIR=/dev

to be:

local DEVDIR=/proc/cygdrive/`cygpath -m /dev | sed 's/://'`

That presumes that "sed" and "cygpath" are available, and that using the dev dir's real-filesystem path would avoid the new /dev interference. (I don't have a broken version of cygwin to try that out.)

If you decide to use something like that, be sure to add a sanity-check of the resulting value, since you don't want the upcoming rm -rf code removing things from a wonky path.

[stepro]

Ok, I've confirmed with Cygwin folks that they will not have the problem, because their setup explicitly creates the directory prior to running post-install scripts. Their recommendation is for the Git for Windows setup to create the directory.

@dscho, I can try to figure this out, but it will take some time and probably isn't going to be the most efficient use of time. If someone else knows exactly where in the InnoSetup stuff directories are defined to be created, and is already knowledgeable on how to test setup, it would go much faster.

[dscho]

If someone else knows exactly where in the InnoSetup stuff directories are defined to be created

I bet you can just add it to the [Dirs] list in our installer definition: https://github.com/git-for-windows/build-extra/blob/b53ba0e5f0d60143a57e9b28ba434c66208901d5/installer/install.iss#L119-L120

For the portable Git, I think you will want to create it just like /tmp/: https://github.com/git-for-windows/build-extra/blob/b53ba0e5f0d60143a57e9b28ba434c66208901d5/portable/release.sh#L76-L77

For MinGit, it would probably be here: https://github.com/git-for-windows/build-extra/blob/b53ba0e5f0d60143a57e9b28ba434c66208901d5/mingit/release.sh#L64-L65

For the NuGet package, I think you'd need to add mkdir -p "$BUILDEXTRA/empty" || die "Could not create an empty directory", say, just before https://github.com/git-for-windows/build-extra/blob/b53ba0e5f0d60143a57e9b28ba434c66208901d5/nuget/release.sh#L82, and then add a line like

    <file src="$buildextra$\nuget\empty\" target="tools\dev" />

after https://github.com/git-for-windows/build-extra/blob/b53ba0e5f0d60143a57e9b28ba434c66208901d5/nuget/GitForWindows.nuspec.in#L26.

Funnily enough, the .tar.bz2 archives are already handled: https://github.com/git-for-windows/build-extra/blob/b53ba0e5f0d60143a57e9b28ba434c66208901d5/archive/release.sh#L51-L53

To test, you can download and install Git for Windows' SDK (caveat: will take a while), then call sdk cd build-extra followed by ./<dir>/release.sh 0-test where <dir> is installer, portable, mingit, etc

Of course, for good measure I would actually create not only the /dev/ directory, but also /dev/mqueue/ and /dev/shm/.

And maybe, just maybe, it would be possible to create the /dev/fd symlink in the installer (and maybe even in portable Git?), too. In Cygwin/MSYS2, those are not _real_ symlinks (because Cygwin still has code that intended to support Windows versions older than Vista, i.e. without actual symlink support) but instead small files marked with the system bit (which _may_ be persisted in .7z archives such as the portable Git self-extracting 7-Zip archive) and the file contents start with the fixed string (in ASCII): !<symlink>, followed by the bytes 0xff and 0xfe and then the symlink target in UTF-16. When I pipe the file contents through hexdump -C, I see this:

00000000  21 3c 73 79 6d 6c 69 6e  6b 3e ff fe 2f 00 70 00  |!<symlink>../.p.|
00000010  72 00 6f 00 63 00 2f 00  73 00 65 00 6c 00 66 00  |r.o.c./.s.e.l.f.|
00000020  2f 00 66 00 64 00 00 00                           |/.f.d...|
00000028

It should be not a problem at all to write this file at the beginning of the post-install actions, via SaveStringToFile() (see e.g. how /etc/install-options.txt is written, and then to flip the system bit, via something like this:

function GetFileAttributes(lpFileName: PAnsiChar): DWORD;
 external '[email protected] stdcall';

function SetFileAttributes(lpFileName: PAnsiChar; 
   dwFileAttributes: DWORD): BOOL; 
external '[email protected] stdcall';

procedure SetSystemBit(FileName : String);
var
 Attr : DWord;
begin
  Attr := GetFileAttributes(FileName);
  if not (Attr and 4) = 4  then          
  begin
    Attr := Attr or 4;
    SetFileAttributes(FileName,Attr);
  end;
end;

@stepro please do help. I already cut into my relax time just researching these pointers. I know you can get it done.

[stepro]

@dscho thanks for all the detailed information. I had no idea there were so many places to change! I can take a look. Cygwin creates /dev, /dev/shm and /dev/mqueue but not the fd files - they leave that to post-install. Even /dev/shm and /dev/mqueue don't need to be created prior to post-install, so for the sake of keeping things simple I'll probably make the smallest change I can to get things working, at least at first, to minimize possible regression elsewhere.

[stepro]

I submitted a PR (git-for-windows/build-extra#255).

[PhilipOakley]

I think... that the PR's message (git-for-windows/build-extra#255) should include a /fixes #2291 line so that they get linked. Hopefully dscho will be able to confirm in due course.

[crypto-rsa]

I've just installed version 2.24.0.windows.1 and the "no such file or directory" problem is back. After some digging, I've found out that the symlinks in /dev are a bit strange. The file /dev/fd contains this: !<symlink>y?/proc/self/fd, i.e. after the !<symlink> part there are bytes 79 3F instead of FF FE as indicated in @dscho's answer above. When I fix these two bytes, it works correctly.

[dscho]

The file /dev/fd contains this: !<symlink>y?/proc/self/fd, i.e. after the !<symlink> part there are bytes 79 3F instead of FF FE as indicated in @dscho's answer above. When I fix these two bytes, it works correctly.

This indeed looks like it is the same issue as described in https://github.com/git-for-windows/git/issues/2388.

Please do consider using the next pre-release cycles to test for things like this, _before_ they hit a full release. Thanks.