Dep: Chmod fails on Windows

[F21]

What version of Go (go version) and dep (git describe --tags) are you using?

Go 1.9-Beta1 and dep c79b048 on Windows 10 64-bit

What dep command did you run?

dep init

My projects are in D:/Work/. My GOPATH is also D:/Work.

The dependency causing issues is https://github.com/prometheus/procfs . It contains folders and files with @colon@ in the path: https://github.com/prometheus/procfs/tree/master/sysfs/fixtures.src/devices/pci0000_%40colon%40_00/0000_%40colon%40_00_%40colon%40_0d.0

What did you expect to see?

init should work correctly without errors.

What did you see instead?

init failed with this error:

safe write of manifest and lock: rename fallback failed: cannot rename C:\Users\francis\AppData\Local\Temp\dep915246291\vendor to D:\Work\src\F21\some_project\vendor: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying file failed: chmod D:\Work\src\F21\some_project\vendor\github.com\prometheus\procfs\sysfs\fixtures.src\devices\pci0000_@colon@_00\0000_@colon@_00_@colon@_0d.0\ata4\host3\target3_@colon@_0_@colon@_0\3_@colon@_0_@colon@_0_@colon@_0\block\sdb\bcache\dirty_data: The system cannot find the path specified.

Inspecting the vendor folder of the project, I see that while D:\Work\src\F21\some_project\vendor\github.com\prometheus\procfs\sysfs\fixtures.src\devices\pci0000_@colon@_00\0000_@colon@_00_@colon@_0d.0\ata4\host3\target3_@colon@_0_@colon@_0\3_@colon@_0_@colon@_0_@colon@_0\block\sdb\bcache\dirty_data exists, a lot of folders and files under D:\Work\src\F21\some_project\vendor\github.com\prometheus\procfs\sysfs\fixtures.src are missing.

[ibrasho]

This issue happened with this version https://github.com/prometheus/procfs/tree/f13c2fcbe05e5f76f69fe0d7e3eabc6679347033/sysfs/fixtures.src

Reading https://github.com/prometheus/procfs/issues/57 should give some context.

[ibrasho]

What does echo %PATH% return in Git Bash and CMD?

[F21]

This is from our slack conversation:

This is the path in git bash:

```$ echo $PATH
/c/Users/user/bin:/mingw64/bin:/usr/local/bin:/usr/bin:/bin:/mingw64/bin:/usr/bin:/c/Users/user/bin:/c/Python27:/c/Python27/Scripts:/c/ProgramData/Oracle/Java/javapath:/c/Php:/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/c/Program Files (x86)/ImageMagick-6.8.4-Q16:/c/WINDOWS/system32:/c/WINDOWS:/c/WINDOWS/System32/Wbem:/c/WINDOWS/System32/WindowsPowerShell/v1.0:/c/Program Files/Calibre2:/c/Program Files (x86)/PuTTY:/c/Program Files (x86)/Windows Kits/8.1/Windows Performance Toolkit:/c/Program Files (x86)/GNU/GnuPG/pub:/c/HashiCorp/Vagrant/bin:/c/Program Files/PuTTY:/c/Program Files (x86)/Skype/Phone:/c/Program Files (x86)/Yarn/bin:/c/Program Files/Git LFS:/c/Go/bin:/cmd:/c/Program Files/nodejs:/d/10-Workspace/go/bin:/bin:/c/Program Files/mingw-w64/x86_64-5.3.0-posix-seh-rt_v4-rev0/mingw64/bin:/c/Protoc/bin:/c/Users/user/AppData/Local/Microsoft/WindowsApps:/c/Users/user/AppData/Local/Microsoft/WindowsApps:/c/Minikube:/c/Users/user/AppData/Local/Yarn/bin:/c/Users/user/AppData/Roaming/npm:/usr/bin/vendor_perl:/usr/bin/core_perl

This is the path in cmd (using cmder)

```λ echo %PATH%
C:\Users\user\Desktop\cmder\bin;C:\Users\user\Desktop\cmder\vendor\conemu-maximus5\ConEmu\Scripts;C:\Users\user\Desktop\cmder\vendor\conemu-maximus5;C:\Users\user\Desktop\cmder\vendor\conemu-maximus5\ConEmu;C:\Python27\;C:\Python27\Scripts;C:\ProgramData\Oracle\Java\javapath;C:\Php;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files (x86)\ImageMagick-6.8.4-Q16;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\Program Files\Calibre2\;C:\Program Files (x86)\PuTTY;C:\Program Files (x86)\Windows Kits\8.1\Windows Performance Toolkit\;C:\Program Files (x86)\GNU\GnuPG\pub;C:\HashiCorp\Vagrant\bin;C:\Program Files\PuTTY\;C:\Program Files (x86)\Skype\Phone\;C:\Program Files (x86)\Yarn\bin;C:\Program Files\Git LFS;C:\Go\bin;C:\Program Files\Git\cmd;C:\Program Files\nodejs\;D:\10-Workspace\go\bin;C:\Program Files\Git\bin;C:\Program Files\mingw-w64\x86_64-5.3.0-posix-seh-rt_v4-rev0\mingw64\bin;C:\Protoc\bin;C:\Users\user\AppData\Local\Microsoft\WindowsApps;C:\Users\user\AppData\Local\Microsoft\WindowsApps;C:\Minikube;C:\Users\user\AppData\Local\Yarn\bin;C:\Users\user\AppData\Roaming\npm;C:\Program Files\Git\usr\bin;C:\Program Files\Git\usr\share\vim\vim74;C:\Users\user\Desktop\cmder\

I am using my dev machine, so the GOPATH is D:/10-workspace/go.

[ibrasho]

I hope we've found the root cause and it doesn't seem to be directly related to dep.

@F21 seems to have core.symlinks set to true in git config which is might be causing this issue.

Currently, %GOPATH%\src\github.com\prometheus\procfs\fixtures\26231\exe is getting created as %GOPATH%\src\github.com\prometheus\procfs\fixtures\26231\exe.symlink and thus we are getting the The system cannot find the path specified. error.

I would recommend the following to debug this further:

• Delete %GOPATH%\github.com\prometheus\procfs.
• git config --unset-all core.symlinks
• go get -u -v github.com/prometheus/procfs
• Check if the %GOPATH%\src\github.com\prometheus\procfs\fixtures\26231\exe file exists in Explorer.
• Run the following code to check if we can access the file with different file separators:

package main

import (
    "fmt"
    "os"
    "path/filepath"
)

func main() {
    path := "D:\\10-workspace\\go\\src\\github.com\\prometheus\\procfs\\fixtures\\26231\\exe"
    if _, err := os.Open(path); err != nil {
        fmt.Println(err)
    }
    if _, err := os.Open(filepath.ToSlash(path)); err != nil {
        fmt.Println(err)
    }
}

Assuming %GOPATH% = D:\10-workspace\go.

[F21]

I am still seeing the same problem.

Here's what I did:

$ git config --global --unset-all core.symlinks
 # no output

$ git config --global --get-all core.symlinks
# no output

$ git config --system --get-all core.symlinks
# no output

$ go get -u -v github.com/prometheus/procfs
github.com/prometheus/procfs (download)
github.com/prometheus/procfs/xfs
github.com/prometheus/procfs

The file %GOPATH%\src\github.com\prometheus\procfs\fixtures\26231\exe still exists.

Running the code produced the following output:

open D:\14-workspace\go\src\github.com\prometheus\procfs\fixtures\26231\exe: The system cannot find the path specified.
open D:/14-workspace/go/src/github.com/prometheus/procfs/fixtures/26231/exe: The system cannot find the path specified.

[ibrasho]

Could you check if %GOPATH%\src\github.com\prometheus\procfs\fixtures\26231\exe still has a .symlink extension?

[F21]

If I turn on file extensions in explorer, it does not have a .symlink extension. But under Type, the file type is listed as .symlink.

[ibrasho]

Was this also the case before switching core.symlinks off?

[F21]

Yes, I before switching it off, the Type is .symlink, and the symlink file had no extension.

[ibrasho]

So I spent a while yesterday installing Windows 10 and trying to see what issues will face me. I'll document these issues here for future reference.

• I used Git Bash (and Cmder).
• Git core.symlinks is disabled (the default).
• I created a new project that imports github.com/prometheus/procfs only. Since it was updated, this issue no longer happens when running dep init.
• I updated the constraint in Gopkg.toml to version = "f13c2fcbe05e5f76f69fe0d7e3eabc6679347033" and ran dep ensure -v and I got an error.

I was getting unable to access /very/long/path/name: Filename too long errors. Turns out you have to enable long path support to enable file paths longer than ~260 characters. After this change (and a restart) I was still getting the same error. I did some research and found that Git config core.longpaths must be set to true.

After these changes, I was able to run dep ensure and vendor github.com/prometheus/procfs@f13c2fcbe05e5f76f69fe0d7e3eabc6679347033 without an error.

I deleted .\vendor and GOPATH\pkg\dep\sources\https---github-com-prometheus-procfs and tried to use dep again when core.symlinks is true.

[F21]

@ibrasho Thanks for taking the time to look into it!

I noticed that it still fails with long path support enabled and core.longpaths enabled with a clean cache through the following steps (only tried in cmder/cmd):

1. Turn on long path support in Windows and reboot.
2. Enable core.longpaths and check the config:

$ git config --system --show-origin core.longpaths

$ git config --global --show-origin core.longpaths
file:C:/Users/user/.gitconfig        true

$ git config --show-origin core.longpaths
file:C:/Users/user/.gitconfig        true

1. Edit Gopkg.tml to and set revision = "f13c2fcbe05e5f76f69fe0d7e3eabc6679347033".

2. Delete the Gopkg.lock file and vendor folder in the project.

3. Delete the dep cache %GOPATH%/pkg/dep

4. Run dep ensure -v:

λ dep ensure -v
Root project is "github.com/F21/test-long-paths"
 1 transitively valid internal packages
 1 external packages imported from 1 projects
(0)   ✓ select (root)
(1)     ? attempt github.com/prometheus/procfs with 1 pkgs; 3 versions to try
(1)         try github.com/prometheus/procfs@f13c2fcbe05e5f76f69fe0d7e3eabc6679347033
(1)     ✓ select github.com/prometheus/procfs@f13c2fcbe05e5f76f69fe0d7e3eabc6679347033 w/2 pkgs
  ✓ found solution with 2 packages from 1 projects

Solver wall times by segment:
  b-source-exists: 4.4650573s
      b-list-pkgs: 1.9307407s
           b-gmal: 1.8297284s
      select-root:   1.0006ms
          satisfy:    500.1µs
         new-atom:    499.7µs
  b-list-versions:         0s
            other:         0s
      select-atom:         0s

  TOTAL: 8.2275268s

grouped write of manifest, lock and vendor: rename fallback failed: cannot rename C:\Users\francis\AppData\Local\Temp\dep270792279\vendor to D:\10-workspace\go\src\github.com\F21\test-long-paths\vendor: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying file failed: chmod D:\10-workspace\go\src\github.com\F21\test-long-paths\vendor\github.com\prometheus\procfs\sysfs\fixtures.src\devices\pci0000_@colon@_00\0000_@colon@_00_@colon@_0d.0\ata4\host3\target3_@colon@_0_@colon@_0\3_@colon@_0_@colon@_0_@colon@_0\block\sdb\bcache\dirty_data: The system cannot find the path specified.

If you do not clear the cache and run dep ensure -v again, it works, but it's because the clone without the strange files is in the cache.

[ibrasho]

I'm already clearing the dep cache and dep ensure -v is succeeding with no errors.

I'd love to know more about your setup. Disk partitions layout, file systems and anything else you think might help. If anyone else has a Windows machine and can try this, please do and report your results.

[F21]

Tested against dep master and still seeing the problem.

In my case, my temp dir is located in C:/. My GOPATH is D:/10-workspace. The dep cache is cleared before running dep ensure. All my disks are NTFS.

[ibrasho]

I'm still puzzled by how the ("The system cannot find the path specified") error could happen or what it means.

I have one last hypothesis before giving in to not call os.Chmod under Windows.

Could you try adding the following code in internal/fs.go and trying dep again? I don't have high hopes for this one but it's worth a try.

[F21]

This is what I get after applying the patch:

$ dep ensure -v
Root project is "github.com/F21/test-long-paths"
 1 transitively valid internal packages
 1 external packages imported from 1 projects
(0)   ✓ select (root)
(1)     ? attempt github.com/prometheus/procfs with 1 pkgs; at least 1 versions to try
(1)         try github.com/prometheus/procfs@f13c2fcbe05e5f76f69fe0d7e3eabc6679347033
(1)     ✓ select github.com/prometheus/procfs@f13c2fcbe05e5f76f69fe0d7e3eabc6679347033 w/2 pkgs
  ✓ found solution with 2 packages from 1 projects

Solver wall times by segment:
  b-source-exists: 7.6619559s
      b-list-pkgs: 1.9702456s
           b-gmal: 1.8392295s
      select-root:    999.8µs
      select-atom:    499.7µs
         new-atom:    499.7µs
            other:         0s
          satisfy:         0s

  TOTAL: 11.4734302s

grouped write of manifest, lock and vendor: rename fallback failed: cannot rename D:\10-Workspace\go\src\github.com\F21\test-long-paths\Gopkg.lock to C:\Users\user\AppData\Local\Temp\dep574806391\Gopkg.lock.orig: copying file failed: close C:\Users\user\AppData\Local\Temp\dep574806391\Gopkg.lock.orig: file already closed

The dependencies do not install correctly and the vendor folder is missing from the project.

[ibrasho]

Oh, sorry I missed that. You will need to delete these lines too before that works.

[F21]

Here are the results after updating those lines and delete the dep cache:

$ dep ensure -v
Root project is "github.com/F21/test-long-paths"
 1 transitively valid internal packages
 1 external packages imported from 1 projects
(0)   ✓ select (root)
(1)     ? attempt github.com/prometheus/procfs with 1 pkgs; 3 versions to try
(1)         try github.com/prometheus/procfs@f13c2fcbe05e5f76f69fe0d7e3eabc6679347033
(1)     ✓ select github.com/prometheus/procfs@f13c2fcbe05e5f76f69fe0d7e3eabc6679347033 w/2 pkgs
  ✓ found solution with 2 packages from 1 projects

Solver wall times by segment:
  b-source-exists: 4.7470922s
      b-list-pkgs: 1.8612316s
           b-gmal:  1.783723s
         new-atom:    500.1µs
      select-atom:    500.1µs
      select-root:    499.7µs
  b-list-versions:         0s
          satisfy:         0s
            other:         0s

  TOTAL: 8.3935467s

grouped write of manifest, lock and vendor: rename fallback failed: cannot rename C:\Users\user\AppData\Local\Temp\dep099666571\vendor to D:\10-Workspace\go\src\github.com\F21\test-long-paths\vendor: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying directory failed: copying file failed: chmod D:\10-Workspace\go\src\github.com\F21\test-long-paths\vendor\github.com\prometheus\procfs\sysfs\fixtures.src\devices\pci0000_@colon@_00\0000_@colon@_00_@colon@_0d.0\ata4\host3\target3_@colon@_0_@colon@_0\3_@colon@_0_@colon@_0_@colon@_0\block\sdb\bcache\dirty_data: The system cannot find the path specified.

[ibrasho]

Ok, I give up. 😩 I"m not able to replicate this locally and all my guesses of why this error could happen have failed. I suggest that we drop os.Chmod for Windows until we figure out while this error is happening.

I would love to hear about it if more people are facing this same issue on Windows.

[ibrasho]

Just gave this another attempt and I believe I found the cause and it's not pretty. 💀

Chmod on Windows doesn't handle long paths (>=248). Other filesystem calls implemented in os use os.fixLongPath to "fix" the path (check file_windows.go).

According to https://msdn.microsoft.com/en-us/library/windows/desktop/aa365247(v=vs.85).aspx#maxpath, this should not be needed since:

Starting in Windows 10, version 1607, MAX_PATH limitations have been removed from common Win32 file and directory functions. However, you must opt-in to the new behavior.

And:

These are the file management functions that no longer have MAX_PATH restrictions if you opt-in to long path behavior: CopyFileW, CopyFile2, CopyFileExW, CreateFileW, CreateFile2, CreateHardLinkW, CreateSymbolicLinkW, DeleteFileW, FindFirstFileW, FindFirstFileExW, FindNextFileW, GetFileAttributesW, GetFileAttributesExW, SetFileAttributesW, GetFullPathNameW, GetLongPathNameW, MoveFileW, MoveFileExW, MoveFileWithProgressW, ReplaceFileW, SearchPathW, FindFirstFileNameW, FindNextFileNameW, FindFirstStreamW, FindNextStreamW, GetCompressedFileSizeW, GetFinalPathNameByHandleW.

GetFileAttributesW & SetFileAttributesW are used in Chmod.

Even though I've already opted-in, these calls still fail if the logic in fixLongPath is not applied to the path. I copied fixLongPath and used it to "fix" the path before passing it to fs.copyFile and things worked smoothly.

I'll open an issue in golang/dep later with more details and I might actually submit the CL.

@sdboyer How should we deal with this in dep? Skip Chmod on windows until this is fixed? Or apply a local fix?

[sdboyer]

@ibrasho 🎉 🎉 that you get your CL in 😄

given that it's in, and it'll be part of 1.9...well, i'm inclined to just leave it as-is and encourage people to update to Go 1.9 if they experience the problem. But, i'll leave it up to you, i'd say you understand the implications of doing that, or compensating for it in dep, much better than i do.

[ibrasho]

🎉 🎉 🎉

I don't have a strong opinion here tbh. If we want to fix it in dep the following needs to be done:

• copy fixLongPath and the other needed unexported funcs: volumeName and isAbs to internal/fs.
• call fixLongPath on dst in copyFile if we are on Windows.

We can add build tags and create multiple versions of copyFile depending on GOOS and the version (to keep this fix < 1.9) but I don't think that's necessary since fixLongPath is idempotent so it's fine if it's called twice.

[mccarty82]

I am suggesting to try "Long Path Tool" program.

[sdboyer]

@ibrasho yeah, we want to fix in dep, but I'm fine with the lazier path of not tagging by version and relying on idempotence

[ibrasho]

I have a feeling I missed something but I'm not sure what. 😅

[sdboyer]

@ibrasho are there possible long-term harms of making the change and getting it wrong?