Git: Git clone does not work in Windows Docker container in mounted host volume folder

[StefanScherer]

• [x] I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

Step 5/10 : RUN git --version --build-options
 ---> Running in 3d9c46cb4a76
git version 2.11.0.windows.1
sizeof-long: 4
machine: x86_64

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

Step 6/10 : RUN cmd.exe /c ver
 ---> Running in 2c465adb3865

Microsoft Windows [Version 10.0.14393]

• What options did you set as part of the installation? Or did you choose the
  defaults?

Step 7/10 : RUN type 'C:\Program Files\Git\etc\install-options.txt'
 ---> Running in ded3512e22b7
Path Option: CmdTools
SSH Option: OpenSSH
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Disabled
Enable Builtin Difftool: Disabled

• Any other interesting things about your environment that might be related
  to the issue you're seeing?

Running Git for Windows in a Windows Container directly on a volume mount point.

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

PowerShell

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

Running inside a Windows container works fine:

PS C:\> docker run git-for-windows-issue git clone https://chromium.googlesource.com/external/gyp
Cloning into 'gyp'...

Running in a Windows container with a volume mounted from the Windows Docker host does not work:

PS C:\> docker run -v "$(pwd):C:\work" git-for-windows-issue git clone https://chromium.googlesource.com/external/gyp
Cloning into 'gyp'...
fatal: Could not switch to '/ContainerMappedDirectories/': No such file or directory

The Container directory C:\work is mounted from the Docker host. This can be shown with the dir command like this:

PS C:\> docker run -v "$(pwd):C:\work" git-for-windows-issue cmd /c dir ..
 Volume in drive C has no label.
 Volume Serial Number is 464C-9AB1

 Directory of C:\

11/22/2016  02:45 PM             1,894 License.txt
07/16/2016  05:18 AM    <DIR>          PerfLogs
12/24/2016  02:12 AM    <DIR>          Program Files
07/16/2016  05:18 AM    <DIR>          Program Files (x86)
12/24/2016  02:12 AM    <DIR>          Users
12/24/2016  02:12 AM    <DIR>          Windows
12/24/2016  02:06 AM    <SYMLINKD>     work [\\?\ContainerMappedDirectories\E5252A88-1539-401E-96FA-005C640978DB]
               1 File(s)          1,894 bytes
               6 Dir(s)  21,271,777,280 bytes free

It seems to me that Git is converting the UNC path into something and tries to switch to the "directory" ContainerMappedDirectories.

Running in a Windows container with a volume mounted from the Windows Docker host works if I use a sub directory below that mount point:

PS C:\> cat .\test-subdir.ps1
if (!(Test-Path subdir)) {
  mkdir subdir
}
cd subdir
git clone https://chromium.googlesource.com/external/gyp

PS C:\> docker run -v "$(pwd):C:\work" git-for-windows-issue powershell -file .\test-subdir.ps1


    Directory: C:\work


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----       12/24/2016   2:25 AM                subdir
Cloning into 'gyp'...
Checking out files: 100% (1645/1645), done.

# back on the host:

PS C:\> dir .\subdir\


    Directory: C:\vagrant\resources\dockerfiles-windows\git-for-windows-issue\subdir


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----       12/24/2016   2:26 AM                gyp

• What did you expect to occur after running these commands?

  Git clone should work directly in the mounted volume directory. I encountered that problem trying to build and test libuv in a container on a mapped volume. They have a command to clone into a sub directory git clone https://chromium.googlesource.com/external/gyp build/gyp but that also doesn't work if current dir in the container is the volume mount point.

• What actually happened instead?

fatal: Could not switch to '/ContainerMappedDirectories/': No such file or directory

• If the problem was occurring with a specific repository, can you provide the
  URL to that repository to help us with testing?

No specific repo. But I have put this test scenario at https://github.com/StefanScherer/dockerfiles-windows/tree/master/git-for-windows-issue with the build and test scripts.

To build the Windows Docker image this Dockerfile is needed

FROM microsoft/windowsservercore

SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPreference = 'SilentlyContinue';"]

ENV chocolateyUseWindowsCompression false

RUN iex ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1')); \
    choco install -y git -params "/GitAndUnixToolsOnPath"

RUN git --version --build-options
RUN cmd.exe /c ver
RUN type 'C:\Program Files\Git\etc\install-options.txt'

# these install-options.txt files do not exist
# RUN type 'C:\Program Files (x86)\Git\etc\install-options.txt'
# RUN type ('{0}\AppData\Local\Programs\Git\etc\install-options.txt' -f $env:USERPROFILE)
# RUN cat /etc/install-options.txt

RUN mkdir C:\work
WORKDIR C:\\work

and then build the Docker image with

PS C:\> docker build -t git-for-windows-issue .

[dscho]

Thank you for this very thorough report.

I won't be near my laptop until next year, so I can only hope that other developers can reproduce the issue in the meantime.

One request from my side before that, though: we fixed a UNC path issue recently and I wonder whether it magically fixes the bug you reported, too. Would you mind testing again with https://github.com/git-for-windows/git/releases/tag/v2.11.1.windows-prerelease.1 ?

[StefanScherer]

Thanks. I've tried the v2.11.1.windows-prerelease.1, but there is the same error message.

No hurry, Docker on Windows is all new. Interesting enough I started to investigate a problem in Node.js which is very similar https://github.com/nodejs/node/issues/8897, digging down to libuv and the build script there tries to clone a git repo and so I found this problem ;-)

[dscho]

Okay. Would you mind pinging me after Jan 2nd so this issue does not fall off my radar?

[vcarluer]

I got the same issue. It happens when I try to clone into a mounted volume or a subdirectory of it.

[asmagin]

I have the same issue. In my case, this is Jenkins that is trying to check-out code before a build. My JENKINS_HOME folder is mounted as a volume.

[StefanScherer]

There is a workaround for programs that want to retrieve the real path of a file or directory. Use a mapped drive letter of that volume and the program stops at the drive letter without stumbling at the volume reparse point.

See an example in my slides:

http://stefanscherer.github.io/windows-docker-workshop/#104

[dscho]

The culprit seems to be that DeviceIoControl() (which we use to parse the target of the symlink) returns a struct whose substitute name (which is supposedly the target) starts with a single backslash, not with the \\?\ prefix. No idea how to get that prefix (we could use the print name of course, but that is specifically intended for displaying, not for substituting...).

[StefanScherer]

The \\?\ only means that it is a long path name that may exceed the MAX_PATH. See https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx for details.

[dscho]

The \?\ only means that it is a long path name that may exceed the MAX_PATH.

It means that (we use that in core.longpaths support), but it means a lot more than that, too.

However, please note that in this case, it is a red herring. There is no such path... the output of dir shows this:

[...]
04/04/2017  10:29 PM    <SYMLINKD>     work [\\?\ContainerMappedDirectories\2AE72BC5-0AFB-4D2B-8A57-49F06FDF77C0]
[...]

However, the link target as per DeviceIoControl() reports \ContainerMappedDirectories\2AE72BC5-0AFB-4D2B-8A57-49F06FDF77C0 (note the missing \\? prefix). Worse: this is obviously not the path we want to use instead of C:\work.

[dscho]

And here is a related docker/docker ticket: https://github.com/docker/docker/issues/31089

[StefanScherer]

Just found this comment https://go-review.googlesource.com/c/41834/ in a commit to fix it for Golang.

Currently windows Stat uses combination of Lstat and Readlink to
walk symlinks until it reaches file or directory. Windows Readlink
is implemented via Windows DeviceIoControl(FSCTL_GET_REPARSE_POINT, ...)
call, but that call does not work on network shares or inside of
Docker container (see issues #18555 ad #19922 for details).

But Raymond Chen suggests different approach:
https://blogs.msdn.microsoft.com/oldnewthing/20100212-00/?p=14963/

• he suggests to use Windows I/O manager to dereferences the
  symbolic link.

This appears to work for all normal symlinks, but also for network
shares and inside of Docker container.

[dscho]

Interesting. So the concrete take-home message is: we should override strbuf_realpath() with a version that uses CreateFile() followed by GetFinalPathNameByHandleW()?

We already have code in compat/mingw.c using that function, and we take care to lazy-load it because backwards compatibility. Although I see that it is supported from Vista up: https://msdn.microsoft.com/en-us/library/windows/desktop/aa364962(v=vs.85).aspx, and it seems as if we simply kept that lazy-load code even after dropping XP support...

So that looks like a viable route forward. It would be really awesome if somebody interested in this issue could take a stab at it (hint, hint) ;-)

[StefanScherer]

I did a short test with ()today and put a example C++ app into my dockerfiles-windows repo.
It seems that GetFinalPathNameByHandleW() does not what we would like to have. In case of a filename/dirname that is on the mounted volume it just returns 0 with GetLastError() = 53 :-(

See https://github.com/StefanScherer/dockerfiles-windows/blob/master/realpath/README.md for some details.

Golang's fix is just for their stat() function, they don't want to retrieve the real path of a file/directory.

[romerod]

Same behavior with git init:
'"C:\Program Files\Git\bin\git.exe" init' command failed.
exit code: 128
stderr: fatal: Invalid path '/ContainerMappedDirectories': No such file or directory

But it doesn't work in a subdir either:
dir C:\BuildAgent
05/10/2017 04:36 PM work [\?ContainerMappedDirectories\A5CD0170-60F5-49DE-B169-05F42A0D8B2C]

mkdir test
cd test
C:\buildAgent\work\test>"C:\Program Files\Git\bin\git.exe" init
fatal: Invalid path '/ContainerMappedDirectories': No such file or directory

[dscho]

@romerod Thank you! However, re-confirmation is no longer necessary at this stage. Help with fixing it would be much appreciated, though.

[dscho]

FWIW I can corroborate @StefanScherer 's findings, at least partially: If I run this

#define _WIN32_WINNT 0x0600
#include <stdio.h>
#include <windows.h>

int main(int argc, char **argv)
{
    int ret = 0;
    HANDLE h = CreateFileW(L".", 0, FILE_SHARE_READ | FILE_SHARE_WRITE |
        FILE_SHARE_DELETE, NULL, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS,
        NULL);
    wchar_t buf[1024];

    if (h == INVALID_HANDLE_VALUE) {
        fprintf(stderr, "Not a valid handle\n");
        return 1;
    }
    if (!GetFinalPathNameByHandleW(h, buf, 1024, 0)) {
        fprintf(stderr, "Error %d\n", (int)GetLastError());
        ret = 1;
    } else
        printf("result: '%S'\n", buf);
    CloseHandle(h);

    return 0;
}

in a directory mapped into Docker, funnily enough I get an ERROR_FILE_NOT_FOUND (which https://msdn.microsoft.com/en-us/library/windows/desktop/aa364962(v=vs.85).aspx does not even list as valid error, the most similar one is ERROR_PATH_NOT_FOUND...).

This is on Windows 10 Enterprise, Creators' Update.

[StefanScherer]

Well I took a look how to help and installed the Git for Windows SDK in a Win2016 VM.

Also tried to put all this into a Windows container with a Dockerfile in C:\git-sdk-64 folder and building an image with it.

# escape=`
FROM microsoft/windowsservercore
COPY . /git-sdk-64
WORKDIR /git-sdk-64
RUN setx /M PATH %PATH%;c:\git-sdk-64\mingw64\bin
# CMD [ "C:\\git-sdk-64\\git-bash.exe" ]
CMD [ "cmd.exe" ]

But most of the tools just won't start, maybe the containerized setup is not fully correct, missing variables or registry keys - I haven't figured out everything yet.
But I know that there are problems with MSYS2 in containers not showing anything in an interactive session in the terminal:

• https://github.com/docker/for-win/issues/262
• https://github.com/StefanScherer/dockerfiles-windows/issues/30

So this is my setup right now:

The installer already comiled git and the test binaries on the Win2016 Docker host, now I also have the binaries in a Windows Container image.
I'm trying to figure out some basics with Sysinternals Process Monitor procmon.exe.
The git.exe itself works in an interactive container and shows eg. the usage on stdout, so I don't give up :-)

Which code would be relevant to investigate, how to recompile only the minimal parts?
I don't know how much time I can spend, but I want to give it a try.

[StefanScherer]

Wait, wait, wait. I just rebuilt my test Docker image described in https://github.com/StefanScherer/dockerfiles-windows/tree/master/git-for-windows-issue and ran the tests and all tests are working correctly.

Step 5/9 : RUN git --version --build-options
 ---> Running in d7e00f193c89
git version 2.13.0.windows.1
built from commit: eba7af3dbb4c846c6303c5f64102acee696c9ab0
sizeof-long: 4
machine: x86_64
 ---> cd4efbccf7e0
Removing intermediate container d7e00f193c89
Step 6/9 : RUN cmd.exe /c ver
 ---> Running in d3ab93a1a2b8

Microsoft Windows [Version 10.0.14393]
 ---> f0b8a00eb335
Removing intermediate container d3ab93a1a2b8
Step 7/9 : RUN type 'C:\Program Files\Git\etc\install-options.txt'
 ---> Running in bd35120b069d
Path Option: CmdTools
SSH Option: OpenSSH
CURL Option: OpenSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Performance Tweaks FSCache: Enabled
Use Credential Manager: Enabled
Enable Symlinks: Disabled

My environment: Windows Server 2016 with all updates installed until revision 1198, Git-for-windows 2.13.0 in a Windows container, Docker 17.05.0-ce.

$(gp 'HKLM:\SOFTWARE\Microsoft\Windows NT\CurrentVersion').BuildLabEx
14393.1198.amd64fre.rs1_release_sec.170427-1353

Repeated the test with git 2.11.0 and still have the error:

PS C:\dockerfiles-windows\git-for-windows-issue> docker run -v "$(pwd):C:\work" git:2.11.0 git clone https://chromium.googlesource.com/external/gyp
Cloning into 'gyp'...
fatal: Could not switch to '/ContainerMappedDirectories/': No such file or directory

So it seems that there already is a fix in git-for-windows 2.13.0 👍

[StefanScherer]

PS C:\dockerfiles-windows\git-for-windows-issue> docker run -v "$(pwd):C:\work" git:2.13.0 git clone https://chromium.googlesource.com/external/gyp
Cloning into 'gyp'...
PS C:\dockerfiles-windows\git-for-windows-issue> dir gyp


    Directory: C:\dockerfiles-windows\git-for-windows-issue\gyp


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        5/11/2017  11:10 PM                buildbot
d-----        5/11/2017  11:10 PM                data
d-----        5/11/2017  11:10 PM                infra
d-----        5/11/2017  11:10 PM                pylib
d-----        5/11/2017  11:10 PM                samples
d-----        5/11/2017  11:10 PM                test
d-----        5/11/2017  11:10 PM                tools
-a----        5/11/2017  11:10 PM              7 .gitignore
-a----        5/11/2017  11:10 PM            491 AUTHORS
-a----        5/11/2017  11:10 PM            250 codereview.settings
-a----        5/11/2017  11:10 PM            533 DEPS
-a----        5/11/2017  11:10 PM            248 gyp
-a----        5/11/2017  11:10 PM            201 gyp.bat
-a----        5/11/2017  11:10 PM           7248 gyptest.py
-a----        5/11/2017  11:10 PM            463 gyp_main.py
-a----        5/11/2017  11:10 PM           1502 LICENSE
-a----        5/11/2017  11:10 PM              3 OWNERS
-a----        5/11/2017  11:10 PM           3691 PRESUBMIT.py
-a----        5/11/2017  11:10 PM            212 README.md
-a----        5/11/2017  11:10 PM            555 setup.py

[romerod]

Result "git init" with 2.13:

C:\buildAgent\work\test>"C:\Program Files\Git\bingit.exe" init
fatal: Invalid path '/ContainerMappedDirectories': No such file or directory

[dscho]

Yes, we know that this fails, and we know in the meantime why it fails, too: the mapped directory is represented as a symbolic link and the target is not accessible as-is, so we cannot resolve it.

The question is not whether it breaks at this point. We know it breaks. The question is how to go about fixing it.

We could reimplement the entire strbuf_realpath() (and actually, I tried that, based on GetFinalPathByHandle(), but as @StefanScherer pointed out, that fails, too), but I am hesitant because it would mean an enormous maintenance burden for me.

We could try to adjust the code in compat/mingw.c to account in lstat() for the fact that we know this /ContainerMappedDirectories and not mark it as a symbolic link in that case. But that means we would have to open the reparse point with every lstat() call on every symbolic link, just to find out that we do not want to handle it as a symbolic link.

We could also patch readlink() to remember when it opened a path that turned out to be a "symbolic link" starting with /ContainerMappedDirectories, return the input path instead, and the next mingw_lstat() should then look whether it should maybe mark the symbolic link as a regular directory.

Or we could hide that reparse reading inside mingw_lstat() in a config value that Docker users have to enable. Ugly, as it requires additional knowledge about Git for Windows, just for Docker users.

I still hope that somebody comes up with a brilliant idea how to detect that we're running in Docker, or even enumerate the mapped directories and mark them as non-symlinks, or some such. That would provide a much better user experience.

Anybody?

[StefanScherer]

Thanks @dscho sharing all these ideas. I also hesitate to dig deeper at the moment as I also see no simple solution. I have talked with some at Microsoft about this pain point as it seems a lot of other languages / frameworks / tools have the same problem getting the real path. What I have heard is that is won't be fixed in Win2016. But now you can think, will Microsoft fix this in another server release in the future, fixing some API's, making them easier to use etc.? I don't know. But at some point they surprise us with new features and then a massive code change in git-for-windows may be not needed. So for now I see others use the drive mapping workaround which works in Java / Jenkins / Bonobo Git Server etc. - maybe that would help us until we have more information if this will be fixed in the OS / API level or must be changed in all 3rd party tools.

[AndriiNikitin]

Similar way none msys2/mingw/cygwin/git-bash tool can handle docker-mounted path.
e.g. :

$ ls -la /cygdrive/c/xxx
lrwxrwxrwx 1 SYSTEM SYSTEM 75 Jun 28 05:44 /cygdrive/c/xxx -> /cygdrive/c/ContainerMappedDirectories/1537DEDA-D28A-4E94-924D-1C6457644DB0
$ cd /cygdrive/c/xxx
bash: cd: /cygdrive/c/xxx: No such file or directory

Is it exactly the same problem or just related?

[romerod]

I'm a bit confused @StefanScherer didn't you comment on the 12. of May that the following works:
PS C:\dockerfiles-windows\git-for-windows-issue> docker run -v "$(pwd):C:\work" git:2.13.0 git clone https://chromium.googlesource.com/external/gyp
?

Eitherway, I tried the workaround with teamcity, I've set the workdir to be G:\ and mapped the volume to C:\buildAgent\work.

The clone part works, thanks at the genius who found that solution. But unluckily "git reset" fails with:
fatal: This operation must be run in a work tree. https://github.com/git/git/blob/maint/setup.c#L393

Anyone found a solution to that? /cc @JetBrains

[romerod]

Ok, I looked a bit into the code and it's calling chdir which calls https://github.com/git-for-windows/git/blob/v2.12.2.windows.2/compat/mingw.c#L679-L693 (didn't verify that).

And using CreateFile/GetFinalPathNameByHandle with the substitution results in error 2 (realpath.exe)

I also tried to use different methods in realpath.exe, by using GetFileInformationByHandleEx(hFile, FileNameInfo, FileInformation, size) I get:

The final path is: \VSMB-{dcc079ae-60ba-4d07-847c-3493609c0870}\c__code

Any ideas?

[StefanScherer]

@romerod Tried again with git 2.15.0.windows.1 in a windowsservercore:1709, but still no luck

PS C:\Users\vagrant\test> docker run -v "$(pwd):C:\work" git-for-windows-issue git clone https://chromium.googlesource.com/external/gyp
Cloning into 'gyp'...
fatal: This operation must be run in a work tree
warning: Clone succeeded, but checkout failed.
You can inspect what was checked out with 'git status'
and retry the checkout with 'git checkout -f HEAD'

I haven't time to look into the git source tree.

[thecloudtaylor]

On a hunch I tried using the same fix for https://github.com/git-for-windows/git/issues/1403 of using mingit (based on @dscho recommendation) and it worked! and it works on Nanoserver as well (bonus!).

````
FROM microsoft/windowsservercore:1709

ENV

ENV MINGET_DOWNLOAD_URL "https://github.com/git-for-windows/git/releases/download/v2.15.1.windows.2/MinGit-2.15.1.2-busybox-64-bit.zip"

Install MinGit

RUN powershell -Command \
$ErrorActionPreference = 'Stop'; \
(New-Object System.Net.WebClient).DownloadFile('%MINGET_DOWNLOAD_URL%', 'mingit.zip') ; \
Expand-Archive mingit.zip -DestinationPath c:\mingit ; \
Remove-Item mingit.zip -Force

RUN setx PATH "c:\mingit\cmd;%PATH%"

RUN mkdir C:\work
WORKDIR C:\work
````

[StefanScherer]

Thanks @taylorb-microsoft for the heads-up for #1403. I'll give it a try.

[StefanScherer]

Yes, when I use MinGit busybox ZIP then both clone and checkout work.
I've updated my test env and Dockerfile https://github.com/StefanScherer/dockerfiles-windows/pull/254

So it works for my use cases.
Shall I close this issue? How about the other participants?

[dscho]

I think this issue still persists...

[romerod]

Tried my luck with mingit busybox too, unfortunately git init has the same problem.

C:\mingit\cmdgit.exe init

fatal: Invalid path '/ContainerMappedDirectories': No such file or directory

[dscho]

Yep, that's the very same issue I was referring to...

[thecloudtaylor]

I was also just able to verify using the latest Windows Insider build along with the latest Windows Insider container image we did fix this even with stock Git for Windows!

Host: https://www.microsoft.com/en-us/software-download/windowsinsiderpreviewserver
Container Image: https://hub.docker.com/r/microsoft/windowsservercore-insider/

[dscho]

@taylorb-microsoft excellent, thanks for testing! I did not know whether there were Insider builds available with the fix, it is really good to know that there are!

[StefanScherer]

Thanks @taylorb-microsoft for the news. Indeed there is no "ContainerMappedDirectories" link when I map a folder into a container:

$ docker run -it -v C:$(pwd):C:/code microsoft/nanoserver-insider:10.0.17046.1000 cmd

Microsoft Windows [Version 10.0.17046.1000]
(c) 2017 Microsoft Corporation. All rights reserved.

C:\>dir
 Volume in drive C has no label.
 Volume Serial Number is 1225-AF06

 Directory of C:\

01/11/2018  11:04 AM    <DIR>          code
11/19/2017  01:41 AM             1,894 License.txt
11/19/2017  01:42 AM    <DIR>          Users
01/11/2018  11:08 AM    <DIR>          Windows
               1 File(s)          2,406 bytes
               3 Dir(s)  21,272,465,408 bytes free

C:\>

Hope this makes it into next official semi annual release!
I'll have to re-try some other languages like Node.js as well... :-)

[TheDukeDK]

Hello there :-)

So just ran into this issue. Have been working with docker containers on Linux for a couple of years. New job, lots of windows based services. Investigating if we can use windows containers. Building a bamboo build agent as a windows container and hit the wall with this issue :-(

Apparently I need to have a special windows insider build of windows 10 to use the latest windowsservercore-insider image.
OR
I can try to use BusyBox MinGit for the git operations.

What is the recommended work around here? Not sure what I need to do to get a special insiders build on my host and I don't wan't to use BusyBox MinGit to find out it is missing some core git functionality.

[cdhunt]

@praqmaTim You can clone to a non-mapped path and then copy the source into the mapped path.

[romerod]

Or use the subst workaround, we are doing that with teamcity buildagents

---

From: Chris Hunt [e-mail]
Sent: Friday, February 23, 2018 7:58:34 PM
To: git-for-windows/git
Cc: Daniel Romero; Mention
Subject: Re: [git-for-windows/git] Git clone does not work in Windows Docker container in mounted host volume folder (#1007)

@praqmaTimhttps://github.com/praqmatim You can clone to a non-mapped path and then copy the source into the mapped path.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/git-for-windows/git/issues/1007#issuecomment-368105860, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ADEJ3isk0J2s4QDLuiqhfgCvRRKOgafEks5tXwpZgaJpZM4LVNBz.

[donny-dont]

@taylorb-microsoft is there any ETA when a non-insider build will have this fix? I've been wanting to get Drone support in windows and it would be great to not have to code a workaround for Windows.

[dscho]

All interested parties: we have a workaround now, available via the newest snapshot at https://wingit.blob.core.windows.net/files/index.html. Could y'all please test?

[ZCube]

I'm really sorry. I missed when testing and merging.
https://github.com/git-for-windows/git/blob/e6ee2f786d91e1c9f82a6c3611a4601b3e16e607/compat/mingw.c#L3569
should be "readlink(path, buf, sizeof(buf)) >=0 &&"

[dscho]

No worries. I'll fix that up real quick.

[dscho]

There you go: 918fa5c06c9. The next snapshot is building and should be available at https://wingit.blob.core.windows.net/files/index.html in about 30 minutes or so.

[ZCube]

remove "!" before readlink please.

[dscho]

remove "!" before readlink please.

Gaaaaah!

Thank you so much. In yet another 30 minutes or so, a corrected snapshot should be available...

[ZCube]

Goal is almost there !!

            readlink(path, buf, sizeof(buf) >= 0) &&

to

            readlink(path, buf, sizeof(buf)) >= 0 &&

Sorry this was my miss ...
:cry:

[dscho]

Oy vey. Thanks for your patience and diligence, @ZCube!

[ZCube]

It's better than just >=0.
Anyway is there snapshot that used busybox?
My test case has submodule test.
Git init and fetch commands works well.

https://github.com/ZCube/drone-windows-container-test/tree/with-git-for-windows-snapshot

proc "pipeline_clone_0" started
+ git init
Initialized empty Git repository in c:/test/src/github.com/drone/envsubst/.git/
+ git remote add origin https://github.com/drone/envsubst.git
+ git fetch --no-tags --depth=50 origin +refs/heads/master:
From https://github.com/drone/envsubst
 * branch            master     -> FETCH_HEAD
 * [new branch]      master     -> origin/master
+ git reset --hard -q d0876d3176965f9552a611cbd56e24a9264355e6
+ git submodule update --init --recursive
exit status 66
proc "pipeline_clone_0" exited with status 1
pipeline_clone_0 : exit code 1

[dscho]

Anyway is there snapshot that used busybox?

@ZCube Here you are: https://wingit.blob.core.windows.net/files/index.html

[ZCube]

It works well.
My test case is all ok!
Thank you!

E:\Workspace\asdf\drone-windows-container-test\pipeline-test>go install github.com/cncd/pipeline/pipec

E:\Workspace\asdf\drone-windows-container-test\pipeline-test>pipec compile --system-arch windows/amd64
Successfully compiled pipeline.yml to pipeline.json

E:\Workspace\asdf\drone-windows-container-test\pipeline-test>pipec exec
proc "pipeline_clone_0" started
+ git init
Initialized empty Git repository in c:/test/src/github.com/drone/envsubst/.git/
+ git remote add origin https://github.com/drone/envsubst.git
+ git fetch --no-tags --depth=50 origin +refs/heads/master:
From https://github.com/drone/envsubst
 * branch            master     -> FETCH_HEAD
 * [new branch]      master     -> origin/master
+ git reset --hard -q d0876d3176965f9552a611cbd56e24a9264355e6
+ git submodule update --init --recursive
proc "pipeline_clone_0" exited with status 0
proc "pipeline_step_0" started
+ "cd c:\\root"
+ "mkdir c:\\root\\test"


    Directory: C:\root


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        4/26/2018   9:44 PM                test
+ "dir"
d-----        4/26/2018   9:44 PM                test
+ "ls"
d-----        4/26/2018   9:44 PM                test
+ "cd c:\\"
+ "cmd /c dir"
 Volume in drive C has no label.
 Volume Serial Number is 047C-45B9

 Directory of C:\

11/23/2016  07:45 AM             1,894 License.txt
03/05/2018  04:42 AM    <DIR>          PerfLogs
04/26/2018  09:44 PM    <DIR>          Program Files
07/16/2016  10:18 PM    <DIR>          Program Files (x86)
04/26/2018  09:44 PM    <DIR>          root
04/26/2018  09:44 PM    <SYMLINKD>     test [\\?\ContainerMappedDirectories\AB4443A3-4FA0-4F41-806D-180278E58A59]
03/05/2018  04:54 AM    <DIR>          Users
03/05/2018  04:52 AM    <DIR>          Windows
               1 File(s)          1,894 bytes
               7 Dir(s)  128,530,272,256 bytes free
+ "dir"


    Directory: C:\


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----         3/5/2018   4:42 AM                PerfLogs
d-r---        4/26/2018   9:44 PM                Program Files
d-----        7/16/2016  10:18 PM                Program Files (x86)
d-----        4/26/2018   9:44 PM                root
d----l        4/26/2018   9:44 PM                test
d-r---         3/5/2018   4:54 AM                Users
d-----         3/5/2018   4:52 AM                Windows
-a----       11/23/2016   7:45 AM           1894 License.txt
+ "ls"
d-----         3/5/2018   4:42 AM                PerfLogs
d-r---        4/26/2018   9:44 PM                Program Files
d-----        7/16/2016  10:18 PM                Program Files (x86)
d-----        4/26/2018   9:44 PM                root
d----l        4/26/2018   9:44 PM                test
d-r---         3/5/2018   4:54 AM                Users
d-----         3/5/2018   4:52 AM                Windows
-a----       11/23/2016   7:45 AM           1894 License.txt
+ "dir c:\\test"


    Directory: C:\test


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        4/26/2018   9:44 PM                src
+ "ls c:\\test"
d-----        4/26/2018   9:44 PM                src
+ "dir c:\\test\\src\\github.com\\drone\\envsubst"


    Directory: C:\test\src\github.com\drone\envsubst


Mode                LastWriteTime         Length Name
----                -------------         ------ ----
d-----        4/26/2018   9:44 PM                parse
-a----        4/26/2018   9:44 PM            185 .drone.yml
-a----        4/26/2018   9:44 PM            519 eval.go
-a----        4/26/2018   9:44 PM           4289 eval_test.go
-a----        4/26/2018   9:44 PM           4186 funcs.go
-a----        4/26/2018   9:44 PM           1587 funcs_test.go
-a----        4/26/2018   9:44 PM           1086 LICENSE
-a----        4/26/2018   9:44 PM            736 README
-a----        4/26/2018   9:44 PM           3372 template.go
-a----        4/26/2018   9:44 PM             18 template_test.go
+ "ls c:\\test\\src\\github.com\\drone\\envsubst"
d-----        4/26/2018   9:44 PM                parse
-a----        4/26/2018   9:44 PM            185 .drone.yml
-a----        4/26/2018   9:44 PM            519 eval.go
-a----        4/26/2018   9:44 PM           4289 eval_test.go
-a----        4/26/2018   9:44 PM           4186 funcs.go
-a----        4/26/2018   9:44 PM           1587 funcs_test.go
-a----        4/26/2018   9:44 PM           1086 LICENSE
-a----        4/26/2018   9:44 PM            736 README
-a----        4/26/2018   9:44 PM           3372 template.go
-a----        4/26/2018   9:44 PM             18 template_test.go


proc "pipeline_step_0" exited with status 0
E:\Workspace\asdf\drone-windows-container-test\pipeline-test>

[sp-jonathan-davis]

@dscho The same error (Invalid path '/ContainerMappedDirectories': No such file or directory) is also occurring with submodule updates. Will your fix also work for submodules?

[ZCube]

@sp-jonathan-davis How about use this Dockerfile?

FROM microsoft/windowsservercore

RUN powershell.exe [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 ; \
    Invoke-WebRequest "https://wingit.blob.core.windows.net/files/MinGit-prerelease-2.17.0.windows.1.36.gdf4ca5fb72-BusyBox-64-bit.zip" \
    -OutFile MinGit-prerelease-2.17.0.windows.1.36.gdf4ca5fb72-BusyBox-64-bit.zip -UseBasicParsing ; \
    Expand-Archive 'MinGit-prerelease-2.17.0.windows.1.36.gdf4ca5fb72-BusyBox-64-bit.zip' -DestinationPath "c:/Git" -Force ; \
    Remove-Item .\MinGit-prerelease-2.17.0.windows.1.36.gdf4ca5fb72-BusyBox-64-bit.zip && \
    setx /m PATH "%PATH%;C:\Git\cmd"

# volume test
VOLUME "C:\\code"
RUN powershell.exe set-itemproperty -path \
    'HKLM:\SYSTEM\CurrentControlSet\Control\Session Manager\DOS Devices' \
    -Name 'V:' -Value '\??\C:\code' -Type String
WORKDIR "V:\\"

Test

Microsoft Windows [Version 10.0.14393]
(c) 2016 Microsoft Corporation. All rights reserved.

V:\>dir c:\
 Volume in drive C has no label.
 Volume Serial Number is 047C-45B9

 Directory of c:\

04/26/2018  11:30 PM    <SYMLINKD>     code [\\?\ContainerMappedDirectories\B5466CAA-AE78-4A16-9C36-1A4B186A3052]
04/26/2018  11:30 PM    <DIR>          Git
11/23/2016  07:45 AM             1,894 License.txt
03/05/2018  04:42 AM    <DIR>          PerfLogs
03/05/2018  04:53 AM    <DIR>          Program Files
07/16/2016  10:18 PM    <DIR>          Program Files (x86)
03/05/2018  04:54 AM    <DIR>          Users
03/05/2018  04:52 AM    <DIR>          Windows
               1 File(s)          1,894 bytes
               7 Dir(s)  128,568,999,936 bytes free

V:\>dir
 Volume in drive V is 1TB
 Volume Serial Number is 8A15-5E5F

 Directory of V:\

04/26/2018  11:30 PM    <DIR>          .
04/26/2018  11:30 PM    <DIR>          ..
               0 File(s)              0 bytes
               2 Dir(s)  88,219,541,504 bytes free

V:\>git clone https://github.com/githubtraining/example-dependency --recursive
Cloning into 'example-dependency'...
remote: Counting objects: 39, done.
remote: Total 39 (delta 0), reused 0 (delta 0), pack-reused 39
Unpacking objects: 100% (39/39), done.
Submodule 'js' (https://github.com/githubtraining/example-submodule.git) registered for path 'js'
Cloning into 'V:/example-dependency/js'...
remote: Counting objects: 18, done.
remote: Total 18 (delta 0), reused 0 (delta 0), pack-reused 18
Submodule path 'js': checked out 'c3c588713233609f5bbbb2d9e7f3fb4a660f3f72'

V:\>cd /d c:\code

c:\code>dir
 Volume in drive C has no label.
 Volume Serial Number is 047C-45B9

 Directory of c:\code

04/26/2018  11:31 PM    <DIR>          .
04/26/2018  11:31 PM    <DIR>          ..
04/26/2018  11:31 PM    <DIR>          example-dependency
               0 File(s)              0 bytes
               3 Dir(s)  88,219,447,296 bytes free

c:\code>git clone https://github.com/githubtraining/example-dependency --recursive test2
Cloning into 'test2'...
remote: Counting objects: 39, done.
remote: Total 39 (delta 0), reused 0 (delta 0), pack-reused 39
Unpacking objects: 100% (39/39), done.
Submodule 'js' (https://github.com/githubtraining/example-submodule.git) registered for path 'js'
Cloning into 'c:/code/test2/js'...
remote: Counting objects: 18, done.
remote: Total 18 (delta 0), reused 0 (delta 0), pack-reused 18
Submodule path 'js': checked out 'c3c588713233609f5bbbb2d9e7f3fb4a660f3f72'

c:\code>exit

[sp-jonathan-davis]

@ZCube Using "MinGit-prerelease-2.17.0.windows.1.36.gdf4ca5fb72-BusyBox-64-bit.zip" I get the following output when trying to clone a project in Jenkins running on a Windows Server 2016 container.

git.exe submodule update --init --recursive lib/hydra

Invalid path '/ContainerMappedDirectories': No such file or directory
clone of '[email protected]:socialpoint/xxx.git' into submodule path 'C:/JENKINS_HOME/xxx/hydra' failed

I wasn't originally able to clone the project so this version is a massive improvement for me. :)

[ZCube]

@sp-jonathan-davis Could you check the version of git? or test with above dockerfile?
It seems mixed versions.

C:\code\example-dependency>git --version
git version 2.17.0.windows.1.36.gdf4ca5fb72

[sp-jonathan-davis]

Using the supplied Dockerfile the submodules are not checked out. When running the submodule update directly I get no output. The equivalent command in a Linux container works as expected.

Edit: I have tested with a freshly created repo with submodules and everything worked. My problem with this specific repo seems to be unrelated to this issue. Thanks for the help!

[dscho]

So can we close this now, thanks to https://github.com/git-for-windows/git/pull/1645?

[mback2k]

I am still encountering the same issue (fatal: this operation must be run in a work tree during git clone) with the preview build from Thu, 3 May 2018 12:46:36 +0200 (commit 05ca542f78).

I am running a container based upon microsoft/windowsservercore:1709 on Windows Server 1709 using process isolation.

The commit seems to have fixed the issue regarding git init, because that works now in mounted host volumes, but git clone / git reset are not working yet.

[StefanScherer]

I think we should try this in the new 1809 images (mcr.microsoft.com/windows/servercore:ltsc2019 mcr.microsoft.com/windows/nanoserver:1809) and and Windows Server 2019 / Windows 10 1809.
The mounted volumes are better than in the ltsc2016 images.

[apetrozzelli]

I think we should try this in the new 1809 images (mcr.microsoft.com/windows/servercore:ltsc2019 mcr.microsoft.com/windows/nanoserver:1809) and and Windows Server 2019 / Windows 10 1809.
The mounted volumes are better than in the ltsc2016 images.

Actually I solved this exact issue moving from ltsc2016 based images to ltsc2019 ones.

[dscho]

I _hope_ this is resolved in the meantime. If not, the ticket is too stale to leave it open.